Check out ProjectPro for DE, DS, ML projects. This link gives you additional discounts: bit.ly/3GmbSq9 Thank you for watching. What are you thoughts on these topics? Let's talk! 😀
Thank you for this helpful video .. My question is that , Are Software Engineers and Data Engineers doing the same work..? because I have not much interested in software development and programming as usual ..SO, Will it be right choice for me to adopt this career as a Data engineer..? Please guide me .. I really appreciate your helpful videos and a lot of Respect from "PAKISTAN".
@@fakhrabashir6471 no software engineering is different. They are more inclined towards web or app development while data engineers develop data pipelines that's generally not exposed to outside users
Linux System Engineer studying Data Science masters, am I right to consider Data Engineer is the natural position I should apply for after I have conquered Hadoop framework , Spark etc ?
As a data engineer, I agree with your points but many of them are barely even negatives, at least for someone who would actually enjoy this kind of job. I enjoy being on the back end and a "second class citizen" as nobody is expecting me to pump out software like they do for the software engineers. The dependencies can get annoying but honestly the investigational work of this job is some of the most fun and keeps you engaged. Learning new software all the time is fun and it's cool to find better ways to do what you were doing before. Overall this job is a privilege to be able to work in as it has amazing WLB, fantastic pay, and low stress compared to 95% of other jobs in the world. Yes it has its challenges but it is nothing compared to other careers, the only hard part really is the initial learning curve.
I'm really interested to become data engineer in future but I really dk which btech degree I choose to pursue first bcuz I'm not that good at studies so Cse degree will be hard but I can get IT or data science
Ive been intrigued into going into data engeneering because im coming from helpdesk and infrastructure engineering. I've beeing trying to get into cloud but its been unsucessful for the past 2 years. I like how you mentioned a decent work life balance as data engineer since its what I'm looking for
The problem with the Business Analyst specialism is [as an intern/junior], employers expect you to have an MBA and/or prior business experience, a solid interest in business/finance in addition to the technical skills, which creates a significant barrier to entry. As a learner looking to change careers at a mature age, data engineering is far more attractive to me, because the emphasis is primarily on the technical skills.
I have all these qualities, still I'm not good enough in India & can't find jobs. Because only CS IT SDE engineers get highpaying jobs in India. We non-tech non-STEM background grads are useless, imagine I have done my BCom, MBA with specialization BA. Can't find a job more than 4 Lakh per annum at max. It's 4700$ per year. I'm tired of this now doing my Masters MS in DS Data Engineering. Hoping to get atleast a fraction of what these SDE software Engineers earn .
@@StellaratorAI wow, sounds like a very tough market in India if you don't have the right background. I think the situation is different in USA, although I'm in a European country. Hopefully your Masters will pay off eventually. Best wishes!
@@gavinkalaher7314 I did my MBA in Business Analytics from USA. doing MS in DS Data Engineering from UK (online) I came back to India from USA, because it's simply not livable viable financially, we could get a better life for just 4000$ in India for what we could get for 150-200k $ in USA. Indian market doesn't pay us or value us.
Isn't the main difference that implementation is different to theoretical understanding? A BA just has to understand the business and tech side, whereas the data eng or data analyst who is in the business have to know their side and be able to build/analyse things. I agree it's a lot to learn though.
Your opening skit got me. My team is basically data engineering, but also core development in our small company too. And once was in a project planning meeting that had some managers, project owners, and marketing/sales in it. The main project was just parsing out text from some documents to check for some sort of risk assessment compliance. And near the end after I just briefly explained my thoughts what doing this would entail for our team which was JUST importing data, parsing for certain identifiers, and generating reports; one of the marketing/sales guys asks, "So when we have this can we advertise this as AI and machind learning because that's big right now". And I basically facepalmed(good thing it was a zoom meeting with my camera off) and got flustered and started say explaining why this wasn't AI and it would be a whole other thing to use that for this project. Which was apparently the wrong thing to say because he just asked why can't we do that? And I'm thinking, "nobody on our team has any experience with that so how can we build that product in your timeframe?" Luckily my boss jumped in and saved me to explain that we couldn't do that now and that it would be a huge time commitment and would derail the whole thing.
Yep it sucks how execs run behind the ML and AI keyword and try to fit it in whenever possible even if it doesn't make sense. Looks like it was a close call 😅
Thank you Jash for this frank video, this puts alot in perspective. I am currently transitioning from Warehouse executive to Data sphere and training to improve soft skills in visualization tool (Power Bi) and query language (MySQL). I end goal was and still is to be a Data Engineer, before now I had no idea these three things you mentioned 1. Second tier folks in most compnay 2. Dependency (bridge between producers and stake holders) and 3. Be Update with technology. This is most appreciated. Thanks also for the link to Project Pro cause I know for a fact, a ceritficate will not get me a job its practice and documentation of work done.
This video is so on point! data engineering is such fast-paced field that by the time you learn something, something else gains momentum and everyone starts following the trend. It is a never-ending process of learning and upgrading.
Yeah us Data Engineers are like doctors who people think, know the answer to all questions. I get question as well like "Why cant you do this? Why are we paying you all this money?" and my answer is usually "This is the job of an architect or a data scientist or ML engineer" and this infuriates them to the core.
I'm looking to move out of data engineering and closer to business decision making. All data roles are slowly becoming one. Data analyst, analytics engineer, data architect, cloud engineer, data engineer, data scientist. ALL of them. Regardless of what the job description says, the company will want you to be able to do everything. I can't take it anymore.
@E H Yes, I'm looking to move OUT of data engineering. I've been a BI Developer, DBA, ETL Developer over the past 20+ years and currently working in the last few years as a data engineer. Over my career, data engineering is really the most unsatisfying role I have had in that is is basically moving data around with little regard to the final business question at hand. I'd rather USE the data to explain business solutions. Data engineering pays more, but is extremely boring.
I'm a business analyst at a small company so some of the data engineering needs will be falling on my shoulders. This was a very informative video with great perspective!
Ohhh so that's why Data engineers move to Data Manager or Data Scientist role after a few year.... I always kept wondering why such good talents choose a different track ... Changing technologies is the most important factor here... unless someone is very passionate about learning new things Data engineering is going to be trouble for them
Salary raises that follow ... Same reason why the SAP was suddenly in fashion in the mid-1990s - salary raises, also, very, very juicy contracts, but the end results as far as I've personally witnessed were rather marginal (I am talking about the overall impact on the industries, say, airplane making - yes, Boeing as the prime example). "Data Engineers" is a 2000 term - prior to it was "Data Stewards" and all these years ever since it was mostly mistaken for DBAs - which is not interchangeable. Other than that, corporate world loves inventing all kinds of titles for no particular merit, mostly for the pay raise, and nothing else. They always rely on the fresh college grads to do all the grunt work, but claim the fruits as theirs. Eventually college grads move onto better paid places and leave all the work to other to handle, hence, average "data engineer" spends a lot of time looking for the better career path - as opposed to bettering his experience by doing his/her work - and that about describes the shallowness of the "data engineers" I've personally interacted/worked with, career hoppers who picked some basic/naive concepts and suddenly deciding they now know everything.
My sister is data engineer and I am very proud of her. She works 10 Hours a day. Some times i feels sad for her for her long working hours and it is taking serious toll on her health.
10 hrs seems tame compared to most other places....9 hrs is the standard anyway, 1 hr extra is almost always expected in corporate. At 12-14 hrs it really starts getting hectic I feel.
as a fellow data engg, working in service based company, i can relate with you. stress is high and WLB is low also pay is less :( we are literally treated as second class citizens and other people will not give us the same respect like SDEs get. also learning part is so much correct. i literally spend some extra hours everyday to learn new tools and tech that i dont get time to spend on other things. starting to regret in data space. its better to be frontend or backend engg who has great pay, respect, high wlb and most imp less learning than DE.
I feel for you brother. I respect data specialists. You are taking care of your family with your professional skills. Your family definitely is proud of you. Ignore all the negative feedback and constantly improve your skills.
Man I completely agree. Im still doing my 4th year and I got an internship in this company . They put me to face the client and expect me to know everything, its not even 3 months in this company. Everyday Im fucking stressed as hell . WLB is fucked . I hate this shit .
So is Data Engineer not something we should do? What area of data is best such as MLE, Data Scientist, Data Engineer, etc., or do you regret working in data itself?
Hi bro I am java developer one of MNC I am planning to switch data engineer role for better pay. What is your opinion switching SDE to DE is good or bad
That has not been my experience as a data engineer, I am getting the same and arguably little better pay as our software developers, and I am more respected as they often come to me and ask for help since I have learned the ins and outs of our data infrastructure so well. I also have less work and stress as I am in the backend and not expected to pump out products like the software engineers. WLB is the same though we all work and leave at the same time.
Hey Jash! I'm a recent high school graduate with a keen interest in pursuing a Computer Science or Information Technology course at an Indian engineering college. Before diving into college, I've decided to take a proactive approach by focusing on learning Java programming and Data Structures and Algorithms (DSA) first. I believe mastering these foundational concepts will provide me with a solid groundwork for my future studies. After gaining proficiency in Java and DSA, my goal is to explore the exciting fields of Data Science and Machine Learning. I'm eager to embark on this learning journey and would appreciate any guidance or advice on the best approach to achieve my goals.
3:35 the Witcher shirt already had my attention but you really had my attention when you said you were a former Zser. I’m a current Zser who’s frantically watching these videos to understand the scope of growth of my role. Literally have a connect with my PD coach this week 😂
Brilliant, making these types of videos help with communication skills, stuff that I have been told I lack of. One thing that you have mentioned is the very broad tech skills we have to deal with, but you haven't said that sometimes it requires to switch from one to another with a snap of a finger. Some pipelines are failing and have to fix them, and at the same time doing your normal development work. Good stuff, I suppose your work is key and haven't been affected by Google laid offs.
lol I have been working as a data engineer and I didn't even know. I created and maintain our current ETL pipeline. It is pretty simple but as I get data from various sources it gets complex. I can relate with the issues that can arise in such cases
Your skit at the beginning was funny. I am an ML Engineer who now works as a Data Engineer. So many people have no idea that the two roles are different things!
Hi Troy, I am also in the process of doing career shift from research ML engineer to Data engineer. As you already did you must be having some idea like pay wise both are equivalent or there is much difference?
@@AnilPawar-jy9sn That is a bit of a difficult question. Theoretically Data Engineers make more. However I took a minor pay cut to secure a job that is more likely to be recession proof and has a management track.
Hi jash, it's difficult to manage DE techstack, companies still don't understand what should they expect from employee, even in interviews they will ask OLTP, OLAP, Hadoop suite of applications, Spark, Snowflake, Databricks with any cloud services, even for them it becomes difficult to chose right candidate.
I liked the topic about "being near and away from the business outcome" I am an ex - frontend developer doing master's in data science now. I have this same question for about 2 years now as I try to figure out my true interest in this big IT industry. So there are 4 factors - 1 salary, 2 significance (how close is our work to the business outcome), 3 dependency (how crucial is our work and to what degree can it drive the stakes), 4 work-life (+ how much can we delegate later in the career) I do not want the best of all. But what I want is maximum value of this fraction: (salary * good work-life) ÷ (dependency or stakes). What role would allow me to maximize this fraction and where will I need to put the 2nd factor significance in this fraction? And somehow dependency and significance also affect number of jobs and their growth rate and salary growths too. What would be the best role if you want the best optimal point from all the above factors? I will consider focusing on that role and it's tech stack and skills.
@@overlord7096 yes. The answer to this question is that it was the wrong mindset to maximize this fraction. The world is vast, I'm going to explore what i feel is right and if there are many options then just pick the ones that pay the most unless they have some sort of a catch that can create a significant problem specifically for you.
@@scrotiemcboogerballs2133 tough question if you want a best personal answer. As of now, I'm going all in for data engineering. And my mental door is open for cloud as well while definitely shut for software development. But the software engineering exposure was worth it, I genuinely feel I have an edge in my learning curve for new technologies in data science as compared to others who directly came to data science. My personal recommendation would be start with data engineering (1st preference), cloud engineering (2nd preference), data analytics (3rd). Then after 3-5 years make data science as top priority with a decent know-how in cloud tech stack.
Excellent video, important points well-presented with real-world examples! The ‘Dependencies’ section (6:00) illustrates why data (or software or most any other type of) engineering will not be eliminated by ChatGPT. AI will get better and better at dealing with the “this ain’t right” complaints, but it will be a while.
I used to think Machine Learning was the brilliant thing in the world of data science, but after doing data science course i have great respect for Data engineers. ❤.
Isn't that true for the DevOps/SRE roles too? Being a DevOps Engineer myself, I always feel like a "Second class citizen" in the sense that my work has little to do with the aspects of the application that are driving the whole business and is mainly about just getting it available to the customer. Sure we're the ones to ensure the availability, redundancy etc and help the organization get rid of mundane manual stuff but i never felt that sense of longing. @Jash Whats your take on this
please make a video covering how to get job at good place with good package as a fresher,companies target,skills,interview prep and what mistakes we should avoid
Well said. Trust me knowledgeable people understand this. Problem is with mid level managers and newbie. They are not clear of this role. Now I am into next stage which is to implement DE techs in Various System Design Architecture.
If that makes you feel better: every employees are considered 2nd class citizens to executives. More seriously, unless it's a tech company, all IT workers are considered 2nd class citizens.
Hi there, I am doing btech computer science from a teir 1.5 college but I have no interest in coding many if my friends have got a software engineer role in big banks like Deutsche and I'm thinking of getting an mba in finance from iim so will I be interior or superior to my engineering batchmates, n will I earn less or more on average, please answer, thanking you
I have been in Data space for more than 1.5 decades. My advice is, if u r not good in math, then pls pls dont go for DS. Bcoz I have been seeing so many primary school mistakes from Data scientists (those who have completed a DS course and claim that they are data scientists). Start ur career with DA/ DE, then parallelly build a good foundation in Math - then slowly switch to ML/AI... As of now, most of the Data scientists are just configuring the Black Boxes (ML algos) - not creating or contributing to the ML algos... I m not kidding... In an interview, I asked a guy to write the equation of a straight line and asked to explain it in simple words... Responses were awesome
I'm trying to become a business analyst for a medium sized company and it seems like it's a hybrid of a regular business analyst and a data engineer/scientist (Knowledge of R, SQL, and other software was a requirement) and I think that's exactly what I want.
Hi Jash, Nice and very informative video. Your consistency of quality is superb!! I am from a service based company working as a DE. I can totally relate to the pain points in service based you were talking about, but that also makes it a thrilling job. Isn't it? I learned today to continuously update myself in the technological aspect. Thanks for that. Please continue bringing this quality content in the future as well!
Thanks Sanidhya! And yes, completely agree. These challenges make data engineering more interesting but some people might not be aware that they exist. Especially newbies. Thanks again though 😊
Yes, totally agree with you. Even I didn't know about these challenges when I joined and was a little bit frustrated, but as I am gaining experience, I am finding it more interesting and challenging. As you mentioned, we have to regularly communicate with our clients, this makes it an intriguing job.
Very precise. By asking to fix the data sources you must rely on different team and if they are slow / unresponsive it may take months to get the results...
For becoming a data engineer do we need to have good knowledge in maths or maths is very important would you clarify this please? Because I'm good at maths but I'm aspiring to become a data engineer and this is the drawback I have
Hi sir..I am from a non technical background... Going through the learning phase... My question is from the the non technical background what are the challenges I can face in the interview...if I can gather the knowledge of the data engineering tools then shall my education matter... Please reply...
All valid points. In cost centers, or CoC, where you are not under the CIO organization, your direct lead is non technical person and you are limited to tools associated with no or limited cost, then you are Data Engineer without described harsh reality, its even worst, all is hyped to Data Science, no one want to invest time to data governance.
Hi Jash Radia, I am From Nepal 🇳🇵. A 19 y/o boy having a curiosity to learn Data Science. Can you make a video on the courses that are available online for data science step-by-step. If you can !! It means a lot to us ❤
Need advice everyone For someone who isn’t a big fan of coding, which role will be most suitable data analyst or data engineer (data scientist is coding algo heavy so not going that side)
Hi, you're are a knowledgeable person. But how to connect you on linkdin? The link is not working. I have few questions, cause I'm beggining my career in related field. So I need help with this. Can you please ?
As a data engineer, I can confirm these points. Migrating to new technology involves a lot of stuff. One of the point that I can think of is the architecture part. As a data engineer, you are often given a responsibility to make architecture that satisfies all the criteria that exist in the client ecosystem. This involves communication between several teams from network to infrastructure to security. You can't go on creating pipelines n all before getting your architecture approved.
Hi Jash, I graduated last year with a Master's degree in Information Systems. I did one job in Database Engineering for ~1 year. Now I’m again hunting job as a Data Engineer. Before January, which certification should I get to show my knowledge when the job market opens in Jan’25? Is Microsoft Azure Data Engineering (DP-203) or Snowflake Data Engineering or any other? What are your thoughts on what I should do this December to make myself eligible for the Data Engineering jobs open in Jan’25?
Thank you for the honest perspectives. Nothing is your solution, everything is your problem (second class). You will know nothing, and you will be happy (tech advances).
Regarding scd type II, how do you explain them to users or business analysts. I had a lot of problems with analysts just using needless transformations to visualise such a tool, as opposed to giving them a long format table that they can visually transform easier. I know that a slider can solve the problem, but I found it that it can take more time to find the solution for them, than just give them what they want and accept my faith. Just curious
I have begineer level of knowledge in Spark. I want to know from where I can prepare or solve interview related Spark questions. Can you provide suggestions for improving my spark skills and any websites or resources to practice?
Currently 5 years of experience have opportunites last atleast I seen some mncs hiring 2-3 years of experience but now job openings are rare I have 3 years of experience in Salesforce development with 5 certifications like pd1, admin, app builder, ai associate and associate, I do worked in 2 projects in my previous company and skills : lwc, Apex, triggers, SOQL, SOSL, integration I'm trying to get job from past 6 months each month I got barely 2 to 3 calls from companies. This market situation is same from past 2023. I need suggestions "I WANT TO TRANSITION FROM SALESFORCE DEV TO DATA ENGINEER" I know learning data engineer skills takes atleast 6 months but compare to salesforce.. job opportunities and growth is more in data engineering. At end of the day every software engineer need money to feed their family. "NEEDED GENUINE SUGGESTIONS PLEASE".
I have been a data analyst for 6 years now, current in a lead role, currently earning over 25 lpa (fixed). Have super high interest in this role but changing careers is pretty daunting. How much would does a DE role get paid for someone starting afresh?
What you mentioned is a good pay for a data analyst. But data engineers earn more than data analysts usually. In consulting firms, I have seen DEs being paid 35 lpa+ for YOE you mentioned and in good tech firms, it can go upto 70-80 lpa+ too. It can go even higher but those are super rare cases.
Hey Jash! i'm shortlisted for the role as a btsa at zs associates. So just wanted to know if there are any similarities in the work that a btsa does to that of what a DE does. Thank you ❤
I'm really interested to become data engineer in future but I really dk which btech degree I choose to pursue first bcuz I'm not that good at studies so Cse degree will be hard but I can get IT or data science
Hi @JashRadia Thank you for the videos. Im trying to find a course where we have a structured system design course which is sufficient for Sr DE interviews.
I'm in 2nd yr and I'm able to solve medium-hard sql ques on Leetcode and stratascratch.... did some 100 python array string ques on gfg and Hackerrank....... By the end of 3rd yr.....I will apply for data roles Now I'm going to learn linux.... My question is :> Is AWS cloud practitioner certificate enough to work on projects..
Great perspective Jash. You definitely laid out the important aspects of the job. One part you missed out was that in many companies you may have to be on call to help with pipeline break/fix if reports/dashboards cater to critical business functions and it is a thankless/low visibility job with nice pay. Google or top tech companies may have SRE teams but that may not be the case with non tech companies. So if you don’t like to do on call/break fixes then don’t be lured by the money. I recently left this field due to on call burnout and I work for a fortune 100 company. Good luck!
I am in data engineer since last 5 years and before that as ETL developer. I feel pretty bored and doesn’t want to learn new technologies. Any suggestions? Want to move out of this. I like make plans more or strategies
Business request changes and so data request will change and complex business logic will have to be implemented. As far as I am concern No-Code Technologies can not meet the expectation of business 100%. There will still be some element of development and embedded coding
Coming from a datawarehouse background and I hear and see DE and DS I remember the need of data architecture and the use of a ETL tool for datawarehousing and BI. DS and DE give me the impression of anything goes and quick and dirty and after a while these projects tend to go to more structured, scd and maybe a need to use an ETL tool to realise that. So I miss a bit the added value of DE and DS over old school datawarehousing.
Excited with your explanation. I am a Solutions Architect. I want to specialize as a Data Engineer. One training Inc is asking 12k for 4 months training. Advice
What about the job security in of a data engineer in product based fintech company when the revenue of company is reducing. So will they fire data engineers or the Java developer.
I worked as a Data Engineer for almost 3 years (Spark application development). I agree with most of the points here. Except few, 1. Data engineers usually have more work compared others like Business Analysts / Data Scientists during the application development. 2. Doing dev work in DE usually comes with some risk. Any bugs in your code has substantially bigger impact. You will be accountable for any errors in data processing. Also it is very difficult thing to track what caused the issue and since when. 3. Testing the correctness of application usually takes longer than other software systems.
I have 4.3 years of experience on GCP, mainly Big query and Composer.. I was laid off last month.. I'm planning to upskill now.. Which skill or tool is the hottest in demand in market right now for me to get a job and also secure next few years of relevance..
I have two years of overall experience as a data engineer... should i stick to it and upskill myself in it or should i slowly learn math and transition to data science? Im really confused. I think I'm interested in both. Please help give suggestions..
Hi jash , I'm a sql developer and im going towards to being a data engineer . My question is how you know what you should learn for now and for future ? Which community you choose as your source for being aware that what is right technologies to learn ?
Most of things I learned didn't come from a community or articles. It came from my own experience that's why I am putting so much efforts into sharing it because it isn't readily available easily
Thank you it was a great informative video, I would like to make a request to understand how a data engineer will approach a problem of the sql database if it has 40-50% is a legacy data and what approach you will take to document it in absence of any formal documents.
Hi Jash. I am a developer and currently I am confused to opt whether Data Engineering or Cloud Computing, Could you please tell which of these two has a better scope and salary in future.
My advice would be to learn cloud fundamentals anyway. Then you need to choose an area that you will work with. Eg, a full stack engineer, data engineer, data scientist etc. This is where you will start focusing more on a particular area. But anything you choose will eventually need atleast fundamental understanding of cloud.
Cloud computing is a subset of data engineering usually. Since most data engineers are cloud data engineers. Hence, they are also well paid compared to only cloud engineers.
Its funny that data engineers feel second class citizen when compared to managers n ds, truth being told all these data centric manager n technical roles have evolved after recent big data boom where companies wants to make most out the data that they have or wants third party company to help them with it. Guess what during the hard times it was these same roles or associated data product teams that were kicked out first, be it during covid pandemic or recent layoffs. If you really want secured n stable job in tech industry you better associate with tech stack onto which the core business runs 24*7, cool tech buzz might give shoot your salary up in corporate world in very short period but it can never ever guarantee stability.
Hi Jash, I've been wanting to choose data science as a career and stumbled across your channel. It's been really amazing and informative learning from you. I am 20 years old in college and have already started learning python after watching your roadmap video. Anything I should be beware of for the future scope in this industry?
I have added some points about future of DE on my QnA video but apart from that, I can't think of anything else for now. Edit: thank you for watching my videos and supporting the channel!
Jash i am interested in DE but i am in big confusion to choose this field.lot of people are going in to this and lot of are coming so it difficult to stay in this field for a longer period? .
Then which is the best domain to work in? I am interning as a DE and all these points kinda demotivated me to leave this ASAP and switch to SDE/Developer roles...
Some people commented on this video that after watching this, they think they made the right choice by choosing DE. But if you think the opposite, then yes. You might want to look for alternatives of DE. It works differently for everyone.
@@JashRadia I don't feel like being a second fiddle of any project or company, and the data word stays within me just because of the AI ML boom currently. Maybe I'll like transitioning into a Data Scientist in future so starting with DE can be a good call as well.
Can you comment on project pro.. Does it really help.. Does it have real time projects which industry uses, what did u feel personally and do you recommend it
Yes! W3 schools first and then leetcode. You can also check this out: sql.datainterview.com/?_ga=2.236460673.1818051070.1674281388-327042888.1671558230&_gl=1*1xpuvg1*_ga*MzI3MDQyODg4LjE2NzE1NTgyMzA.*_ga_DWYL646T38*MTY3NDI4MTM4OC44LjEuMTY3NDI4MTM5NC4wLjAuMA..
I have been working as Data scientist for 2 years but not getting much work on Data science things instead I get other things. Recently I started working on DE tools it caught my interest, was planning to switch from DS to DE What are your suggestions?
the amount of times i had talks with C-level managers expecting something like a datalake solving aaaall their analytical problems ... completely oblivious about it's not more than a tool .. enabling fast processing of large data amounts and grab and forget data collection .. but it's not providing value .. (yet) all the work has to be done anyways ..no matter which tool is used ..
Hi @JashRadia I'm working in QA as automation testing I have opted course for Data engineering Do you think good idea to switch from QA to DE? Are pays are better in DE then QA?
Hi i am trying to switch my career from mechanical to data engineering Can you sugesstes please for the streaming which tool to learn is apache kafka is enough ?
Thanks for the video Jash.. you literally spoke my mind about DE being just the support function in Product Based and captive Industry. There is always a cap on budget and most of the stuff we do as a DE goes under the rug as the work wont get enough visibility to stakeholders.
Data engineer vs AI engineer, which will be the better path for someone trying to switch career after 6-7 years of experience as software developer role. Please talk about future opportunities and job market
Hi Jash i am good at writing sql queries and have keen interest in data engineering,, coming to coding part I can do easy and manage things but feel demotivated when can not solve difficult questions like software engineer s do i am looking to make long career in this what do you suggest,,,
It is okay. Data engineering doesn't require difficult coding solving skills. Basic to intermediate is enough. It shouldn't stop you from coming in this field. Just focus on one language and try to solve 1-2 questions everyday on leetcode. You will automatically be better after a few months
Hi sir.. I have 3 years of experience in Production support and started taking course on ADE but I will on my maternity leave till sep 2023..Can I switch to ade telling 3 years of experience can you please suggest me in this sir
Hey great video, helpful information. I currently work as Data Scientist for 2 years, I thinking of exploring Data Engineer career path, what should be my initial learning in terms of technology or tool?
actually the first point is a good thing to me being a second class citizen because we don't have to face so much questions that the non technical stake holders ask and we work at the backend and we don't getting attention like the mle or ds get and also have a good pay
Check out ProjectPro for DE, DS, ML projects. This link gives you additional discounts: bit.ly/3GmbSq9
Thank you for watching. What are you thoughts on these topics? Let's talk! 😀
Thank you for this helpful video .. My question is that , Are Software Engineers and Data Engineers doing the same work..? because I have not much interested in software development and programming as usual ..SO, Will it be right choice for me to adopt this career as a Data engineer..? Please guide me .. I really appreciate your helpful videos and a lot of Respect from "PAKISTAN".
Do not forget that if a new director is coming you might get demoted/fired
@@fakhrabashir6471 no software engineering is different. They are more inclined towards web or app development while data engineers develop data pipelines that's generally not exposed to outside users
@@JashRadia Thank you so much.. your guidance always give me a right direction towards my career. May God bless you with healthy, wealthy life..❤️
Linux System Engineer studying Data Science masters, am I right to consider Data Engineer is the natural position I should apply for after I have conquered Hadoop framework , Spark etc ?
As a data engineer, I agree with your points but many of them are barely even negatives, at least for someone who would actually enjoy this kind of job. I enjoy being on the back end and a "second class citizen" as nobody is expecting me to pump out software like they do for the software engineers. The dependencies can get annoying but honestly the investigational work of this job is some of the most fun and keeps you engaged. Learning new software all the time is fun and it's cool to find better ways to do what you were doing before. Overall this job is a privilege to be able to work in as it has amazing WLB, fantastic pay, and low stress compared to 95% of other jobs in the world. Yes it has its challenges but it is nothing compared to other careers, the only hard part really is the initial learning curve.
I'm really interested to become data engineer in future but I really dk which btech degree I choose to pursue first bcuz I'm not that good at studies so Cse degree will be hard but I can get IT or data science
Very well put. For those who really like DE, this video will only help them like it more I guess 😅
Ive been intrigued into going into data engeneering because im coming from helpdesk and infrastructure engineering. I've beeing trying to get into cloud but its been unsucessful for the past 2 years. I like how you mentioned a decent work life balance as data engineer since its what I'm looking for
can we connect?, I want to break into this career I have a lot of doubts , would appreciate your guidance.
The problem with the Business Analyst specialism is [as an intern/junior], employers expect you to have an MBA and/or prior business experience, a solid interest in business/finance in addition to the technical skills, which creates a significant barrier to entry. As a learner looking to change careers at a mature age, data engineering is far more attractive to me, because the emphasis is primarily on the technical skills.
Yes. Totally agreed!
I have all these qualities, still I'm not good enough in India & can't find jobs. Because only CS IT SDE engineers get highpaying jobs in India. We non-tech non-STEM background grads are useless, imagine I have done my BCom, MBA with specialization BA. Can't find a job more than 4 Lakh per annum at max. It's 4700$ per year.
I'm tired of this now doing my Masters MS in DS Data Engineering. Hoping to get atleast a fraction of what these SDE software Engineers earn .
@@StellaratorAI wow, sounds like a very tough market in India if you don't have the right background. I think the situation is different in USA, although I'm in a European country. Hopefully your Masters will pay off eventually. Best wishes!
@@gavinkalaher7314 I did my MBA in Business Analytics from USA. doing MS in DS Data Engineering from UK (online)
I came back to India from USA, because it's simply not livable viable financially, we could get a better life for just 4000$ in India for what we could get for 150-200k $ in USA.
Indian market doesn't pay us or value us.
Isn't the main difference that implementation is different to theoretical understanding? A BA just has to understand the business and tech side, whereas the data eng or data analyst who is in the business have to know their side and be able to build/analyse things.
I agree it's a lot to learn though.
Your opening skit got me. My team is basically data engineering, but also core development in our small company too. And once was in a project planning meeting that had some managers, project owners, and marketing/sales in it. The main project was just parsing out text from some documents to check for some sort of risk assessment compliance. And near the end after I just briefly explained my thoughts what doing this would entail for our team which was JUST importing data, parsing for certain identifiers, and generating reports; one of the marketing/sales guys asks, "So when we have this can we advertise this as AI and machind learning because that's big right now".
And I basically facepalmed(good thing it was a zoom meeting with my camera off) and got flustered and started say explaining why this wasn't AI and it would be a whole other thing to use that for this project. Which was apparently the wrong thing to say because he just asked why can't we do that? And I'm thinking, "nobody on our team has any experience with that so how can we build that product in your timeframe?" Luckily my boss jumped in and saved me to explain that we couldn't do that now and that it would be a huge time commitment and would derail the whole thing.
Yep it sucks how execs run behind the ML and AI keyword and try to fit it in whenever possible even if it doesn't make sense. Looks like it was a close call 😅
Please refer
Thank you Jash for this frank video, this puts alot in perspective. I am currently transitioning from Warehouse executive to Data sphere and training to improve soft skills in visualization tool (Power Bi) and query language (MySQL). I end goal was and still is to be a Data Engineer, before now I had no idea these three things you mentioned 1. Second tier folks in most compnay 2. Dependency (bridge between producers and stake holders) and 3. Be Update with technology. This is most appreciated. Thanks also for the link to Project Pro cause I know for a fact, a ceritficate will not get me a job its practice and documentation of work done.
This video is so on point! data engineering is such fast-paced field that by the time you learn something, something else gains momentum and everyone starts following the trend. It is a never-ending process of learning and upgrading.
Thank you and yeah, well put.
Yeah us Data Engineers are like doctors who people think, know the answer to all questions. I get question as well like "Why cant you do this? Why are we paying you all this money?" and my answer is usually "This is the job of an architect or a data scientist or ML engineer" and this infuriates them to the core.
Ikr! People usually don't notice things done in the backend 🤷♂️
I'm looking to move out of data engineering and closer to business decision making. All data roles are slowly becoming one. Data analyst, analytics engineer, data architect, cloud engineer, data engineer, data scientist. ALL of them. Regardless of what the job description says, the company will want you to be able to do everything. I can't take it anymore.
@@EH-it8pj yes it's too hard. Better to become a business analyst and just work with spreadsheets and let someone else deal with all the coding.
@@JashRadia we face the same thing, people value what analytics doing but not how they are able to do it.
@E H Yes, I'm looking to move OUT of data engineering. I've been a BI Developer, DBA, ETL Developer over the past 20+ years and currently working in the last few years as a data engineer. Over my career, data engineering is really the most unsatisfying role I have had in that is is basically moving data around with little regard to the final business question at hand. I'd rather USE the data to explain business solutions. Data engineering pays more, but is extremely boring.
I'm a business analyst at a small company so some of the data engineering needs will be falling on my shoulders. This was a very informative video with great perspective!
Great to know, thanks!
Ohhh so that's why Data engineers move to Data Manager or Data Scientist role after a few year....
I always kept wondering why such good talents choose a different track ...
Changing technologies is the most important factor here...
unless someone is very passionate about learning new things Data engineering is going to be trouble for them
Salary raises that follow ...
Same reason why the SAP was suddenly in fashion in the mid-1990s - salary raises, also, very, very juicy contracts, but the end results as far as I've personally witnessed were rather marginal (I am talking about the overall impact on the industries, say, airplane making - yes, Boeing as the prime example).
"Data Engineers" is a 2000 term - prior to it was "Data Stewards" and all these years ever since it was mostly mistaken for DBAs - which is not interchangeable.
Other than that, corporate world loves inventing all kinds of titles for no particular merit, mostly for the pay raise, and nothing else. They always rely on the fresh college grads to do all the grunt work, but claim the fruits as theirs. Eventually college grads move onto better paid places and leave all the work to other to handle, hence, average "data engineer" spends a lot of time looking for the better career path - as opposed to bettering his experience by doing his/her work - and that about describes the shallowness of the "data engineers" I've personally interacted/worked with, career hoppers who picked some basic/naive concepts and suddenly deciding they now know everything.
My sister is data engineer and I am very proud of her. She works 10 Hours a day. Some times i feels sad for her for her long working hours and it is taking serious toll on her health.
10 hrs seems tame compared to most other places....9 hrs is the standard anyway, 1 hr extra is almost always expected in corporate. At 12-14 hrs it really starts getting hectic I feel.
Introductory part was pretty awesome. Good improvements in content presentation. Keep it up so we will also keep supporting you to grow.
Thank you so much!
as a fellow data engg, working in service based company, i can relate with you. stress is high and WLB is low also pay is less :( we are literally treated as second class citizens and other people will not give us the same respect like SDEs get. also learning part is so much correct. i literally spend some extra hours everyday to learn new tools and tech that i dont get time to spend on other things. starting to regret in data space. its better to be frontend or backend engg who has great pay, respect, high wlb and most imp less learning than DE.
I feel for you brother. I respect data specialists. You are taking care of your family with your professional skills. Your family definitely is proud of you. Ignore all the negative feedback and constantly improve your skills.
Man I completely agree. Im still doing my 4th year and I got an internship in this company . They put me to face the client and expect me to know everything, its not even 3 months in this company. Everyday Im fucking stressed as hell . WLB is fucked . I hate this shit .
So is Data Engineer not something we should do? What area of data is best such as MLE, Data Scientist, Data Engineer, etc., or do you regret working in data itself?
Hi bro I am java developer one of MNC I am planning to switch data engineer role for better pay. What is your opinion switching SDE to DE is good or bad
That has not been my experience as a data engineer, I am getting the same and arguably little better pay as our software developers, and I am more respected as they often come to me and ask for help since I have learned the ins and outs of our data infrastructure so well. I also have less work and stress as I am in the backend and not expected to pump out products like the software engineers. WLB is the same though we all work and leave at the same time.
This is the first video of you and I subscribed! What a content and a clear communication 👏👏
Thank you!
Hey Jash! I'm a recent high school graduate with a keen interest in pursuing a Computer Science or Information Technology course at an Indian engineering college. Before diving into college, I've decided to take a proactive approach by focusing on learning Java programming and Data Structures and Algorithms (DSA) first. I believe mastering these foundational concepts will provide me with a solid groundwork for my future studies. After gaining proficiency in Java and DSA, my goal is to explore the exciting fields of Data Science and Machine Learning. I'm eager to embark on this learning journey and would appreciate any guidance or advice on the best approach to achieve my goals.
Your approach is great. Just remember, DSA, SQL and Data Modeling are foundational blocks for any data role.
3:35 the Witcher shirt already had my attention but you really had my attention when you said you were a former Zser. I’m a current Zser who’s frantically watching these videos to understand the scope of growth of my role. Literally have a connect with my PD coach this week 😂
Thank you! And all the best 😊
Brilliant, making these types of videos help with communication skills, stuff that I have been told I lack of.
One thing that you have mentioned is the very broad tech skills we have to deal with, but you haven't said that sometimes it requires to switch from one to another with a snap of a finger. Some pipelines are failing and have to fix them, and at the same time doing your normal development work.
Good stuff, I suppose your work is key and haven't been affected by Google laid offs.
I am yet to know my fate but keeping positive! 😊
lol I have been working as a data engineer and I didn't even know. I created and maintain our current ETL pipeline. It is pretty simple but as I get data from various sources it gets complex. I can relate with the issues that can arise in such cases
Your skit at the beginning was funny. I am an ML Engineer who now works as a Data Engineer. So many people have no idea that the two roles are different things!
Ikr!
Hi Troy, I am also in the process of doing career shift from research ML engineer to Data engineer. As you already did you must be having some idea like pay wise both are equivalent or there is much difference?
@@AnilPawar-jy9sn That is a bit of a difficult question. Theoretically Data Engineers make more. However I took a minor pay cut to secure a job that is more likely to be recession proof and has a management track.
How to become ML engineer??
Hi jash, it's difficult to manage DE techstack, companies still don't understand what should they expect from employee, even in interviews they will ask OLTP, OLAP, Hadoop suite of applications, Spark, Snowflake, Databricks with any cloud services, even for them it becomes difficult to chose right candidate.
Yes true. It does get overwhelming. But key is to show that you are good at skills you know and show confidence that you will learn the rest.
I liked the topic about "being near and away from the business outcome" I am an ex - frontend developer doing master's in data science now. I have this same question for about 2 years now as I try to figure out my true interest in this big IT industry. So there are 4 factors -
1 salary,
2 significance (how close is our work to the business outcome),
3 dependency (how crucial is our work and to what degree can it drive the stakes),
4 work-life (+ how much can we delegate later in the career)
I do not want the best of all. But what I want is maximum value of this fraction: (salary * good work-life) ÷ (dependency or stakes). What role would allow me to maximize this fraction and where will I need to put the 2nd factor significance in this fraction? And somehow dependency and significance also affect number of jobs and their growth rate and salary growths too.
What would be the best role if you want the best optimal point from all the above factors? I will consider focusing on that role and it's tech stack and skills.
Did you got the answer?
@@overlord7096 yes. The answer to this question is that it was the wrong mindset to maximize this fraction. The world is vast, I'm going to explore what i feel is right and if there are many options then just pick the ones that pay the most unless they have some sort of a catch that can create a significant problem specifically for you.
really love your question and looking for the answer too :3
@@s0ulweaver What career path do you recommend?
@@scrotiemcboogerballs2133 tough question if you want a best personal answer. As of now, I'm going all in for data engineering. And my mental door is open for cloud as well while definitely shut for software development. But the software engineering exposure was worth it, I genuinely feel I have an edge in my learning curve for new technologies in data science as compared to others who directly came to data science. My personal recommendation would be start with data engineering (1st preference), cloud engineering (2nd preference), data analytics (3rd). Then after 3-5 years make data science as top priority with a decent know-how in cloud tech stack.
Then what are the jobs that fits into "go to starbucks and open up the laptop" profile??
too underrated comment
Excellent video, important points well-presented with real-world examples!
The ‘Dependencies’ section (6:00) illustrates why data (or software or most any other type of) engineering will not be eliminated by ChatGPT. AI will get better and better at dealing with the “this ain’t right” complaints, but it will be a while.
I landed a Data engineering Internship and I am hoping it turns into a job after I graduate. The company has been awesome far
All the best!
Please give me a link, thanks
How did you got the internship, can you please share ?
bro I also got internship for 6months. but they extended it for another 3 months. What abt you ?
I used to think Machine Learning was the brilliant thing in the world of data science, but after doing data science course i have great respect for Data engineers. ❤.
As an AI Engineer now, Couldn't agree more!
Isn't that true for the DevOps/SRE roles too? Being a DevOps Engineer myself, I always feel like a "Second class citizen" in the sense that my work has little to do with the aspects of the application that are driving the whole business and is mainly about just getting it available to the customer. Sure we're the ones to ensure the availability, redundancy etc and help the organization get rid of mundane manual stuff but i never felt that sense of longing. @Jash Whats your take on this
I am a data and reporting engineer, and I find I have a lot of control over the operations and even the direction of the business. Be proactive. Lead.
please make a video covering how to get job at good place with good package as a fresher,companies target,skills,interview prep and what mistakes we should avoid
Sure, thanks for suggestion. Will try to make a video around this.
which linux distro you recommend for DS,ML
@@zeusdelta2842 it doesn't matter. All distributions have ways to install commonly used packages.
@@JashRadia still i love to know your preference 😃
@@zeusdelta2842 I usually prefer Ubuntu then cent os. I avoid RHEL since it is expensive to licence.
Well said. Trust me knowledgeable people understand this. Problem is with mid level managers and newbie. They are not clear of this role. Now I am into next stage which is to implement DE techs in Various System Design Architecture.
If that makes you feel better: every employees are considered 2nd class citizens to executives.
More seriously, unless it's a tech company, all IT workers are considered 2nd class citizens.
Hi there, I am doing btech computer science from a teir 1.5 college but I have no interest in coding many if my friends have got a software engineer role in big banks like Deutsche and I'm thinking of getting an mba in finance from iim so will I be interior or superior to my engineering batchmates, n will I earn less or more on average, please answer, thanking you
@@swirlzet1971 have you got some clearity about your career choice?
I have been in Data space for more than 1.5 decades. My advice is, if u r not good in math, then pls pls dont go for DS. Bcoz I have been seeing so many primary school mistakes from Data scientists (those who have completed a DS course and claim that they are data scientists). Start ur career with DA/ DE, then parallelly build a good foundation in Math - then slowly switch to ML/AI... As of now, most of the Data scientists are just configuring the Black Boxes (ML algos) - not creating or contributing to the ML algos...
I m not kidding...
In an interview, I asked a guy to write the equation of a straight line and asked to explain it in simple words... Responses were awesome
Totally true 😅 maths without DS is like kayaking without pedals.
Hi sir
Hi sir how can I interect with you please don't say no
I'm trying to become a business analyst for a medium sized company and it seems like it's a hybrid of a regular business analyst and a data engineer/scientist (Knowledge of R, SQL, and other software was a requirement) and I think that's exactly what I want.
Hi Jash, Nice and very informative video. Your consistency of quality is superb!!
I am from a service based company working as a DE. I can totally relate to the pain points in service based you were talking about, but that also makes it a thrilling job. Isn't it?
I learned today to continuously update myself in the technological aspect. Thanks for that.
Please continue bringing this quality content in the future as well!
Thanks Sanidhya! And yes, completely agree. These challenges make data engineering more interesting but some people might not be aware that they exist. Especially newbies. Thanks again though 😊
Yes, totally agree with you. Even I didn't know about these challenges when I joined and was a little bit frustrated, but as I am gaining experience, I am finding it more interesting and challenging. As you mentioned, we have to regularly communicate with our clients, this makes it an intriguing job.
@@sanidhyajoshi5619i sanidhya, i also work in service based company,I want to discuss something with you regarding DE,can I get your Gmail or contact
Very precise. By asking to fix the data sources you must rely on different team and if they are slow / unresponsive it may take months to get the results...
Yes, exactly!
Great video, really appreciated the insights on the challenges of being a data engineer
Thank you! 😊
Uhmm 10:44, The underlying file format in a delta lake is parquet. It only allows ACID transactions when in a delta lake environment like Databricks
For becoming a data engineer do we need to have good knowledge in maths or maths is very important would you clarify this please? Because I'm good at maths but I'm aspiring to become a data engineer and this is the drawback I have
You do need basic mathematics skills for any CS background field. But you don't need advanced maths in DE unlike in ML
After that opening skit, I love this guy!
Hi sir..I am from a non technical background... Going through the learning phase... My question is from the the non technical background what are the challenges I can face in the interview...if I can gather the knowledge of the data engineering tools then shall my education matter... Please reply...
All valid points. In cost centers, or CoC, where you are not under the CIO organization, your direct lead is non technical person and you are limited to tools associated with no or limited cost, then you are Data Engineer without described harsh reality, its even worst, all is hyped to Data Science, no one want to invest time to data governance.
Yes, so true.
Hi Jash Radia, I am From Nepal 🇳🇵. A 19 y/o boy having a curiosity to learn Data Science. Can you make a video on the courses that are available online for data science step-by-step. If you can !! It means a lot to us ❤
The Intro 🤣👌🏻 Very well made video!!! Great Job Jash... ✌🏻
Thank you! 😁
Need advice everyone
For someone who isn’t a big fan of coding, which role will be most suitable data analyst or data engineer (data scientist is coding algo heavy so not going that side)
Data Analyst, infact i will suggest business analytics
Hi, you're are a knowledgeable person. But how to connect you on linkdin? The link is not working. I have few questions, cause I'm beggining my career in related field. So I need help with this. Can you please ?
As a data engineer, I can confirm these points.
Migrating to new technology involves a lot of stuff.
One of the point that I can think of is the architecture part. As a data engineer, you are often given a responsibility to make architecture that satisfies all the criteria that exist in the client ecosystem. This involves communication between several teams from network to infrastructure to security.
You can't go on creating pipelines n all before getting your architecture approved.
True and I have separate data architects who sit on their ivory tower and never get their hands dirty with implementation
what bothers me the most is multiple tech stack doing the same thing, not getting fit because of it, and the job opportunities are still less.
Hi Jash,
I graduated last year with a Master's degree in Information Systems.
I did one job in Database Engineering for ~1 year.
Now I’m again hunting job as a Data Engineer.
Before January, which certification should I get to show my knowledge when the job market opens in Jan’25?
Is Microsoft Azure Data Engineering (DP-203) or Snowflake Data Engineering or any other?
What are your thoughts on what I should do this December to make myself eligible for the Data Engineering jobs open in Jan’25?
Cool on the point view! Love it :)
Thank you for the honest perspectives.
Nothing is your solution, everything is your problem (second class).
You will know nothing, and you will be happy (tech advances).
Exactly 💯💯 thanks for watching 😁
Regarding scd type II, how do you explain them to users or business analysts. I had a lot of problems with analysts just using needless transformations to visualise such a tool, as opposed to giving them a long format table that they can visually transform easier. I know that a slider can solve the problem, but I found it that it can take more time to find the solution for them, than just give them what they want and accept my faith. Just curious
I have learnt python and it’s libraries for ML I have learnt MYSQL and tableau ! What else is required for a data engineer role
Checkout my data engineering roadmap video for this
as an aspiring data scientist this was very informative
I have begineer level of knowledge in Spark. I want to know from where I can prepare or solve interview related Spark questions. Can you provide suggestions for improving my spark skills and any websites or resources to practice?
Checkout my roadmap video on the channel. In the description, you will find resources that I recommend for learning spark well.
Currently 5 years of experience have opportunites last atleast I seen some mncs hiring 2-3 years of experience but now job openings are rare I have 3 years of experience in Salesforce development with 5 certifications like pd1, admin, app builder, ai associate and associate, I do worked in 2 projects in my previous company and skills : lwc, Apex, triggers, SOQL, SOSL, integration I'm trying to get job from past 6 months each month I got barely 2 to 3 calls from companies. This market situation is same from past 2023. I need suggestions "I WANT TO TRANSITION FROM SALESFORCE DEV TO DATA ENGINEER" I know learning data engineer skills takes atleast 6 months but compare to salesforce.. job opportunities and growth is more in data engineering. At end of the day every software engineer need money to feed their family. "NEEDED GENUINE SUGGESTIONS PLEASE".
I have been a data analyst for 6 years now, current in a lead role, currently earning over 25 lpa (fixed). Have super high interest in this role but changing careers is pretty daunting. How much would does a DE role get paid for someone starting afresh?
What you mentioned is a good pay for a data analyst. But data engineers earn more than data analysts usually. In consulting firms, I have seen DEs being paid 35 lpa+ for YOE you mentioned and in good tech firms, it can go upto 70-80 lpa+ too. It can go even higher but those are super rare cases.
Hey Jash! i'm shortlisted for the role as a btsa at zs associates. So just wanted to know if there are any similarities in the work that a btsa does to that of what a DE does. Thank you ❤
Congratulations! Yes there are so many similarities. Depending on your team your role will be 60-95% similar to a data engineer. Happy learning.
@@JashRadia Thank you so much for the reply sir
@@Jay-yu2jx no problem man! And no need for sir 😊
I'm really interested to become data engineer in future but I really dk which btech degree I choose to pursue first bcuz I'm not that good at studies so Cse degree will be hard but I can get IT or data science
Hi @JashRadia Thank you for the videos. Im trying to find a course where we have a structured system design course which is sufficient for Sr DE interviews.
There is no perfect course for it. But you'll find a lot of RUclips videos on it. Like mock system design interviews
I'm in 2nd yr and I'm able to solve medium-hard sql ques on Leetcode and stratascratch.... did some 100 python array string ques on gfg and Hackerrank.......
By the end of 3rd yr.....I will apply for data roles
Now I'm going to learn linux....
My question is :> Is AWS cloud practitioner certificate enough to work on projects..
I would suggest AWS solutions architect. You can skip the cloud practitioner one.
@@JashRadia okayyy thanku so much
Great perspective Jash. You definitely laid out the important aspects of the job. One part you missed out was that in many companies you may have to be on call to help with pipeline break/fix if reports/dashboards cater to critical business functions and it is a thankless/low visibility job with nice pay. Google or top tech companies may have SRE teams but that may not be the case with non tech companies. So if you don’t like to do on call/break fixes then don’t be lured by the money. I recently left this field due to on call burnout and I work for a fortune 100 company. Good luck!
So true. Thanks for adding this!
I am in data engineer since last 5 years and before that as ETL developer. I feel pretty bored and doesn’t want to learn new technologies.
Any suggestions? Want to move out of this. I like make plans more or strategies
Try to move to analytics engineer role or upskill in ML and move to MLE type of roles.
Business request changes and so data request will change and complex business logic will have to be implemented. As far as I am concern No-Code Technologies can not meet the expectation of business 100%. There will still be some element of development and embedded coding
Yes, so true. It can improve but not to a point where you don't need any development effort.
Awesome !! love you bro.... your content is really helpful.
Coming from a datawarehouse background and I hear and see DE and DS I remember the need of data architecture and the use of a ETL tool for datawarehousing and BI. DS and DE give me the impression of anything goes and quick and dirty and after a while these projects tend to go to more structured, scd and maybe a need to use an ETL tool to realise that. So I miss a bit the added value of DE and DS over old school datawarehousing.
Excited with your explanation. I am a Solutions Architect. I want to specialize as a Data Engineer. One training Inc is asking 12k for 4 months training. Advice
Is it possible to get remote jobs in data engineering for freshers?
Bro, how does Google prescribe Dataflow for new data pipelines and it does not even get a mention from their own engineer?
Is it really a niche?
Dataflow is recommended many times by us GCP engineers to customers. It is not that niche.
What about the job security in of a data engineer in product based fintech company when the revenue of company is reducing.
So will they fire data engineers or the Java developer.
No company who values data will do that. And without data, it's very hard to survive in this age.
I worked as a Data Engineer for almost 3 years (Spark application development). I agree with most of the points here. Except few,
1. Data engineers usually have more work compared others like Business Analysts / Data Scientists during the application development.
2. Doing dev work in DE usually comes with some risk. Any bugs in your code has substantially bigger impact. You will be accountable for any errors in data processing. Also it is very difficult thing to track what caused the issue and since when.
3. Testing the correctness of application usually takes longer than other software systems.
Exactly
Thanks Jash for bringing up much much valid situations and scenarios for data engineering.. 👍
i wanted to choose data science as a career.can you do a daily work life of a data engineer?also wanted to know how regularly communicate with clients
It depends on your company and project. For client meetings, it can happen daily or atleast twice in a week
@@JashRadia thanks
I just found your channel. This is awesome. Can you do a top ten onlilne resources (free or not) for learning DEng?
Checkout the roadmap video on this channel for this.
Hi Jash, Thanks for the info. Which one is good for a Masters student in Data Science, Data Engineer or Data Scientist?
Data scientist. But gaining skills of DE will still be helpful
I have 4.3 years of experience on GCP, mainly Big query and Composer.. I was laid off last month.. I'm planning to upskill now.. Which skill or tool is the hottest in demand in market right now for me to get a job and also secure next few years of relevance..
I think you have discovered my roadmap video already. Hope it helps!
I have a confusion, should I focus more on MySQL or PostgreSql ? I know that most of the syntax are same.
Doesn't matter. You should know one of them. I have seen postgres used a little more than MySQL though.
I have two years of overall experience as a data engineer... should i stick to it and upskill myself in it or should i slowly learn math and transition to data science? Im really confused. I think I'm interested in both. Please help give suggestions..
Hi jash , I'm a sql developer and im going towards to being a data engineer .
My question is how you know what you should learn for now and for future ? Which community you choose as your source for being aware that what is right technologies to learn ?
Most of things I learned didn't come from a community or articles. It came from my own experience that's why I am putting so much efforts into sharing it because it isn't readily available easily
Thank you it was a great informative video, I would like to make a request to understand how a data engineer will approach a problem of the sql database if it has 40-50% is a legacy data and what approach you will take to document it in absence of any formal documents.
Hi Jash. I am a developer and currently I am confused to opt whether Data Engineering or Cloud Computing, Could you please tell which of these two has a better scope and salary in future.
My advice would be to learn cloud fundamentals anyway. Then you need to choose an area that you will work with. Eg, a full stack engineer, data engineer, data scientist etc. This is where you will start focusing more on a particular area. But anything you choose will eventually need atleast fundamental understanding of cloud.
Cloud computing is a subset of data engineering usually. Since most data engineers are cloud data engineers. Hence, they are also well paid compared to only cloud engineers.
Thank you Prasad and Jash🙏🏻
Its funny that data engineers feel second class citizen when compared to managers n ds, truth being told all these data centric manager n technical roles have evolved after recent big data boom where companies wants to make most out the data that they have or wants third party company to help them with it. Guess what during the hard times it was these same roles or associated data product teams that were kicked out first, be it during covid pandemic or recent layoffs. If you really want secured n stable job in tech industry you better associate with tech stack onto which the core business runs 24*7, cool tech buzz might give shoot your salary up in corporate world in very short period but it can never ever guarantee stability.
Hi Jash, I've been wanting to choose data science as a career and stumbled across your channel. It's been really amazing and informative learning from you. I am 20 years old in college and have already started learning python after watching your roadmap video. Anything I should be beware of for the future scope in this industry?
I have added some points about future of DE on my QnA video but apart from that, I can't think of anything else for now.
Edit: thank you for watching my videos and supporting the channel!
Jash i am interested in DE but i am in big confusion to choose this field.lot of people are going in to this and lot of are coming so it difficult to stay in this field for a longer period? .
No. I have seen many in this field for a long time. It is not difficult. People move and and out by their preferences
Then which is the best domain to work in? I am interning as a DE and all these points kinda demotivated me to leave this ASAP and switch to SDE/Developer roles...
Some people commented on this video that after watching this, they think they made the right choice by choosing DE. But if you think the opposite, then yes. You might want to look for alternatives of DE. It works differently for everyone.
@@JashRadia I don't feel like being a second fiddle of any project or company, and the data word stays within me just because of the AI ML boom currently. Maybe I'll like transitioning into a Data Scientist in future so starting with DE can be a good call as well.
How I can move from database developer to data engineer. With an experience of 3.5 yr in(database developer )
Can you comment on project pro.. Does it really help.. Does it have real time projects which industry uses, what did u feel personally and do you recommend it
Can you suggest any good source to master SQL and DSA, I'm planning to switch from testing to DE
Yes! W3 schools first and then leetcode. You can also check this out: sql.datainterview.com/?_ga=2.236460673.1818051070.1674281388-327042888.1671558230&_gl=1*1xpuvg1*_ga*MzI3MDQyODg4LjE2NzE1NTgyMzA.*_ga_DWYL646T38*MTY3NDI4MTM4OC44LjEuMTY3NDI4MTM5NC4wLjAuMA..
hope you were able to switch...
If yes pls let me know how you managed to do so...
I have been working as Data scientist for 2 years but not getting much work on Data science things instead I get other things. Recently I started working on DE tools it caught my interest, was planning to switch from DS to DE
What are your suggestions?
It should be your choice but yes I have seen a few people make this switch if they are interested in more core tech work compared to DS.
the amount of times i had talks with C-level managers expecting something like a datalake solving aaaall their analytical problems ... completely oblivious about it's not more than a tool .. enabling fast processing of large data amounts and grab and forget data collection .. but it's not providing value .. (yet) all the work has to be done anyways ..no matter which tool is used ..
Jash wanted to know if an analyst with 6+ years of experience can transition into data engineer roles?
Yes, this is very common, too. If you want to then go for it!
Hi @JashRadia
I'm working in QA as automation testing
I have opted course for Data engineering
Do you think good idea to switch from QA to DE?
Are pays are better in DE then QA?
Yes pays are better. I have seen a few people I personally know switching from QA to DE
Hi i am trying to switch my career from mechanical to data engineering
Can you sugesstes please for the streaming which tool to learn is apache kafka is enough ?
Kafka is more than enough. Alternatives are AWS kinesis, GCP pubsub etc. But learning any 1 is enough
Thanks for the video Jash.. you literally spoke my mind about DE being just the support function in Product Based and captive Industry. There is always a cap on budget and most of the stuff we do as a DE goes under the rug as the work wont get enough visibility to stakeholders.
Data engineer vs AI engineer, which will be the better path for someone trying to switch career after 6-7 years of experience as software developer role. Please talk about future opportunities and job market
Both are great options but defer a lot. I think my next video will help you choose between some of these roles... It will be uploaded soon
It is very well explained, thanks for your effort 👍
Thank you!
Hi Jash i am good at writing sql queries and have keen interest in data engineering,, coming to coding part I can do easy and manage things but feel demotivated when can not solve difficult questions like software engineer s do i am looking to make long career in this what do you suggest,,,
It is okay. Data engineering doesn't require difficult coding solving skills. Basic to intermediate is enough. It shouldn't stop you from coming in this field. Just focus on one language and try to solve 1-2 questions everyday on leetcode. You will automatically be better after a few months
@@JashRadia that's great thank you jash hope to take things forward.🙌
You may learn aws dbt
Is that a #YNWA in the badckground? and thanks for the insights btw!
Yes that is! Thanks 😁
Sir, how much data structures and algorithms is required for cracking a D.E. interview ?
You need a lot of leetcode easy and medium practice
Thanks much Jash for the details . Iam currently working as oracle dba in rotation shifts.Could you suggest some job with less coding skills
Data analyst is a good job with less coding skills
Hi sir..
I have 3 years of experience in Production support and started taking course on ADE but I will on my maternity leave till sep 2023..Can I switch to ade telling 3 years of experience can you please suggest me in this sir
Yes you can but right now, times are tough to make a job change with shaky economy and job market.
@@JashRadia thanks a lot for replying..I ll try to switch after ML that is nov .. hope it will be safe
Hey great video, helpful information. I currently work as Data Scientist for 2 years, I thinking of exploring Data Engineer career path, what should be my initial learning in terms of technology or tool?
Bro I need job of Data Analyst entry level please help me
Thank you and Check out the data engineering roadmap video on my channel. It will answer your question and more.
actually the first point is a good thing to me being a second class citizen because we don't have to face so much questions that the non technical stake holders ask and we work at the backend and we don't getting attention like the mle or ds get and also have a good pay
Yeah, exactly