0:23 1st mistake: understand the purpose of building the data pipeline, ask the why instead of straight dive into the technical details 2:02 2nd mastering sql is non negotiable 3:08 3rd automate everything 4:36 4th mistake: did not learn about debugging, maybe try Grafana / Prometheus, with an alerting mechanism 5:54 5th Learning is a continuous process
Am a ETL/Data guy for 14 years... trust me he is giving the best advice which no one will say... DEBUGGING man I wish some one would have told me when I was fresher
YOU VIDEOS ARE REALLY AMAZING!! I never comment usually but your work is really inspiring. Data always excites me and being a data research analyst now I wanted to shift to data engineering as I'm in love the work flow of it. Since I knew fundamentals in MYSQL and python for data analysis which includes fundamentals like loops, functions and conditional statements then forwarding to hands on numpy and pandas. Is it enough or is there something I should learn apart from this for data engineering ?
Thank you for great contributions to the community and make me realise to contribute something to the community. So I started recently contributing to the community with SQL interview preparation, problem and solution series. Soon I will contribute many Thank you.
Eagerly waiting for the Fundamentals of Data Engineering Masterclass! PS: Could you please make a video on Microsoft Fabric? Is it helpful as a data engineers? Should we learn this tool?
It starts from asking questions to business people, what are they really expecting and do with the data, then from there you can start thinking about how to execute it
@DarshilParmar ... Is data Engineering is a suitable for someone who is love coding very much and don't like work with visualization stuff like dashboard, who like to focus on distributed systems, scalablility performanece and need to heavy logic thinking to do
Bro in second part of youtube analysis video s3 trigger is not working properly and the data(parquet files) is not getting uploaded and no graph is visible
@@DarshilParmar i checked can it be related to the thing that data types have to be same when creating etl jobs, interface has changed completly,kindly help
Bro how can we create a code snippets like you showed in thumbnail? Which tool it is I saw many people use this to post their code on LinkedIn also, That one with 3 color dots on top left?
Hello guys, Can anyone send me link to the lecture where Darshil sets the AWS pricing alarm. It would be really helpful. I found the where he created the account then IAM user but not the one where he sets the alarm. It is there but I've lost it.
Hi Darshil, I have some experience on Azure adf. This combo will help me to go as a azure data engineer or only for AWS engineers. Have you used Azure cloud in this combo ?
I am struggling to become a Data Engineer. In few months I will write my Bachelor thesis and it should be about data engineering. I hope with this thesis i will be able to enter the job market. Any recommendations for a thesis to enter the market?
Hi bhaiya Needed a laptop in college and programming i started learning data science and ml but dont do gaming so i didnt take any GPU laptop I have 2 options First lenovo ideapad pro 5 with 14" ultra 9 H processor 32 gb ram 1 tb ssd and 84 wh battery Second is lenovo yoga slim 7i with ultra 7 processor 32 gb ram 1 tb ssd and 65 wh battery Both price same Which one should i go for?
Sir can u pls tell me that whether Information and Data Security subject or Essentials of Data Collection and Ethics subject will be the better option for data engineering career. This is an elective subject in my 3rd sem
Hi All, I have one interview question for ADF? If i have load 10 different tables from ADLS to destination. All these tables have different schema. How i ensure each table will insert data for his destination table only? My reply was: Using parameters we can map the tables but i am not sure how i can check schema constraints. Any idea anyone?
I had created an account on 24 June, From today total 8 days are remaining I.e till 25 july for my free trial period. But it gets deactivated and asking my credit card details, WILL THEY CHARGE ME ??
Sir, I'm a third year cse student. I want to be a data engineer and could you please suggest resources for that. And, do freshers get job in this field?
Hi bhaiya Needed a laptop in college I chose 2 options hp pavilion plus 16gb ram fixed 1 tb ram fixed i7 13700h rtx3050 6gb 1.01 lakh Ideapad pro 5 16gb ram fixed 1 tb ssd ultra 7 155H rtx 3050 6gb 1.13 lakh Which one should i prefer and if i want to go more low which can i take
@@DarshilParmar bhaiya actually i am very confused on which laptop to take and dont want to overspend I dont do machine learning now but may do so i dont know if i should go for a dedicated graphics card and which processor to go i7 or ultra 7 and ram Can you please suggest me which configuration should i go for and what price range I am getting lenovo ideapad same laptop with 13700h online for 94k but parents are saying not to buy online so i cant decide what to do but i need a laptop now
@@lectures3727 If you play games then graphic card is good else just go for other option Basic config - 16GB ram, 500-1TB SSD is good, any latest processor. If you can afford maybe try Microsoft surface or macbook
@@DarshilParmar no i dont play games Can i go for any latest processor I am learning data science machine learning now Will that laptop be enough if i take ultra 7 or i7 I am mostly getting p type processor
can we get more discount on your combo course for limited period like it is still expensive for students who are just starting off in the field , hope you understand and you'll do consider it
@@avinashvyas1597 Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Purchase Individual Courses with 50% off: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT50 3. Data Warehouse for Data Engineering with Snowflake - pay.datavidhya.com/cop/BpfGFSahpQ?code=DWT50 4. Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache USE CODE: EARLYSPARK for 50% off
1500 Likes and I will upload Fundamentals of Data Engineering Masterclass here
For free?? Darshil
1500 on the video? Because you are about there in one day or that comment? Y'all smash the like on both, we need our masterclass!
Can u make videos on these topics u have said so that we can understand properly.
@DarshilParmar done, we want the masterclass 🤝
I am among 1.5k .❤
0:23 1st mistake: understand the purpose of building the data pipeline, ask the why instead of straight dive into the technical details
2:02 2nd mastering sql is non negotiable
3:08 3rd automate everything
4:36 4th mistake: did not learn about debugging, maybe try Grafana / Prometheus, with an alerting mechanism
5:54 5th Learning is a continuous process
Am a ETL/Data guy for 14 years... trust me he is giving the best advice which no one will say... DEBUGGING man I wish some one would have told me when I was fresher
DEBUGGING is mostly skipped all are focusing on building while most of the time you to debug.
you deserve more subscribers and likes
YOU VIDEOS ARE REALLY AMAZING!!
I never comment usually but your work is really inspiring.
Data always excites me and being a data research analyst now I wanted to shift to data engineering as I'm in love the work flow of it. Since I knew fundamentals in MYSQL and python for data analysis which includes fundamentals like loops, functions and conditional statements then forwarding to hands on numpy and pandas. Is it enough or is there something I should learn apart from this for data engineering ?
Thank you for great contributions to the community and make me realise to contribute something to the community. So I started recently contributing to the community with SQL interview preparation, problem and solution series. Soon I will contribute many Thank you.
Eagerly waiting for the Fundamentals of Data Engineering Masterclass!
PS: Could you please make a video on Microsoft Fabric? Is it helpful as a data engineers? Should we learn this tool?
How to understand the flow of data for the specific project or work
It starts from asking questions to business people, what are they really expecting and do with the data, then from there you can start thinking about how to execute it
I just Googled that how to like a video 1500 times with single account(Unfortunately No option), Please upload Master class
Will do it soon
Best video editing
Waiting for your Fundamentals of Data Engineering Masterclass 💙
Coming soon
Really Helpful!!!
Thank you
@DarshilParmar ... Is data Engineering is a suitable for someone who is love coding very much and don't like work with visualization stuff like dashboard, who like to focus on distributed systems, scalablility performanece and need to heavy logic thinking to do
Bro in second part of youtube analysis video s3 trigger is not working properly and the data(parquet files) is not getting uploaded and no graph is visible
Check IAM permission
@@DarshilParmar i checked can it be related to the thing that data types have to be same when creating etl jobs, interface has changed completly,kindly help
Great video Darshil...If i carrier change to data engineer to learn is difficult to get job ?
Could you please make a video on Debugging concepts, especially w.r.t. data engineering?
Will do
Can you please upload some thorough projects along with the business mindset behind them and also how we should build further projects
I have lot of them on my RUclips - check RUclips analytics
Thank you Brother!
You are welcome
Bro how can we create a code snippets like you showed in thumbnail? Which tool it is I saw many people use this to post their code on LinkedIn also, That one with 3 color dots on top left?
Just write code to image on google
@@DarshilParmar thank you so much bro!
Thank you Darshil. How to learn about debugging?
Darshil now that you have more than 1.5k likes
Could you please upload the data engineering fundamentals masterclass
I need to build a PPT, record, and edit video, it takes time! will upload once it is done
@@DarshilParmar Hey Thank You for your reply. And Thanks for all the efforts you are taking. I look forward to the masterclass 👍
Is ml ,dl is important in data engineering???
no DE is Another sector
Hey mate , great video. Thank you. Would you let me know what software do you use for video editing
I used to use FinalCutPro but now I have an editor who edits my videos
Which tool are you using to create architecture diagrams?
Google Slides
Hello guys, Can anyone send me link to the lecture where Darshil sets the AWS pricing alarm. It would be really helpful. I found the where he created the account then IAM user but not the one where he sets the alarm. It is there but I've lost it.
@DarshilParmar Hello, tagging you if you can help with the same.
Check cloudwatch - docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
@@DarshilParmar Sure, Thanks!
Hi Darshil, I have some experience on Azure adf. This combo will help me to go as a azure data engineer or only for AWS engineers.
Have you used Azure cloud in this combo ?
Yes, we have project in Spark course on Azure
@@DarshilParmar what about pipeline implementation by using Azure ?
Yes, it's building data pipeline only!
Hi Darshil Make Videos on - Palantir Tool - very trending now in DE world ..
Can you share link?
Timelines for next set of cource launch kafka airflow and projects
can you please share how you create these animations in your videos?
I am struggling to become a Data Engineer. In few months I will write my Bachelor thesis and it should be about data engineering. I hope with this thesis i will be able to enter the job market. Any recommendations for a thesis to enter the market?
I'd say learn high-demand tools, try to get any job in data, doesn't have to be DE only and make your way from there
Hi bhaiya
Needed a laptop in college and programming i started learning data science and ml but dont do gaming so i didnt take any GPU laptop
I have 2 options
First lenovo ideapad pro 5 with 14" ultra 9 H processor 32 gb ram 1 tb ssd and 84 wh battery
Second is lenovo yoga slim 7i with ultra 7 processor 32 gb ram 1 tb ssd and 65 wh battery
Both price same
Which one should i go for?
Sir can u pls tell me that whether Information and Data Security subject or Essentials of Data Collection and Ethics subject will be the better option for data engineering career. This is an elective subject in my 3rd sem
Plz ,anyone can tell me which CSE specialization should i choose for data engineering ??
Hi All,
I have one interview question for ADF?
If i have load 10 different tables from ADLS to destination. All these tables have different schema. How i ensure each table will insert data for his destination table only?
My reply was: Using parameters we can map the tables but i am not sure how i can check schema constraints.
Any idea anyone?
I had created an account on 24 June,
From today total 8 days are remaining I.e till 25 july for my free trial period.
But it gets deactivated and asking my credit card details, WILL THEY CHARGE ME ??
Helpful
Bhaiya how it is data engineering professional certificate by ibm on coursera
I suggest you to do GCP one
@@DarshilParmar and for data analytics and data science how is ibm data analytics and data science course?
I completed my b. Com computers and later i have competed my masters in marketing domain. Is am a eligible for a data engineer
Is dsa compulsary to crack an product based company , if yes then how much , any suggestions from where to get this skill set
I have a video on this ruclips.net/video/gAk9m_d27yk/видео.htmlsi=fJOrV6Gzxa8Ecyov
Sir, I'm a third year cse student. I want to be a data engineer and could you please suggest resources for that. And, do freshers get job in this field?
Check out my roadmap
Hi bhaiya
Needed a laptop in college
I chose 2 options hp pavilion plus 16gb ram fixed 1 tb ram fixed i7 13700h rtx3050 6gb 1.01 lakh
Ideapad pro 5 16gb ram fixed 1 tb ssd ultra 7 155H rtx 3050 6gb 1.13 lakh
Which one should i prefer and if i want to go more low which can i take
Don't go HP
@@DarshilParmar bhaiya actually i am very confused on which laptop to take and dont want to overspend
I dont do machine learning now but may do so i dont know if i should go for a dedicated graphics card and which processor to go i7 or ultra 7 and ram
Can you please suggest me which configuration should i go for and what price range
I am getting lenovo ideapad same laptop with 13700h online for 94k but parents are saying not to buy online so i cant decide what to do but i need a laptop now
@@lectures3727 If you play games then graphic card is good else just go for other option
Basic config - 16GB ram, 500-1TB SSD is good, any latest processor.
If you can afford maybe try Microsoft surface or macbook
@@DarshilParmar no i dont play games
Can i go for any latest processor
I am learning data science machine learning now
Will that laptop be enough if i take ultra 7 or i7
I am mostly getting p type processor
Can a person from outside india . Can take data vidya combo course
Yes you can - datavidhya.com/combo-course-info
@@DarshilParmar and the subscription is for life time
@@hemanthvarupula9873 Yes
can we get more discount on your combo course for limited period like it is still expensive for students who are just starting off in the field , hope you understand and you'll do consider it
I have 50% offer going on, if you want you can start with one course first and then go one by one
@@DarshilParmarI was trying to buy your sql course but there is no discount even the coupon code is also not valid
Is there any other way around ?
@@avinashvyas1597
Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479
USE CODE: COMBO50 for 50% off
Purchase Individual Courses with 50% off:
1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50
2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT50
3. Data Warehouse for Data Engineering with Snowflake - pay.datavidhya.com/cop/BpfGFSahpQ?code=DWT50
4. Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache
USE CODE: EARLYSPARK for 50% off
@@DarshilParmar Thank you Darshil
Also you’ll be mentoring me now is that correct ?
@@avinashvyas1597 There are no 1 to 1 session but we have dedicated discord for this so if you have questions I will help you out
Data Engineer Job Role and AI Contribution: ruclips.net/video/NR7L0TSFtSE/видео.htmlsi=kj4d_8rCdbQSCpPc
did you had a knowledge on data engineering when you started your internship
I had knowledge about ML and Data Science but then I learned DE on internship
How much you making per month
❤
Kya thi cho bhaiband?
Please provide discount in your course
. please provide an email id or whatsapp to connect with you
50% offer is going on, please check description