hey buddy trust me, you are doing just great i watched many videos on youtube almost all creators, but you are gem bro.... just loved the way you are teaching, thanks you brother...
Usually, I don't comment on RUclips videos but the quality of this Azure end-to-end project video is so high that I couldn't resist commenting. This is way better than many fancy paid courses and udemy courses. Keep up the great work Ansh, best wishes!!
I am so happy that my video really helped you a lot. Please subscribe and share it with others. Lots of content is coming to your way every SUNDAY. Stay Tuned!
This project is fantastic, and every step is explained in great detail-truly appreciate the effort! Looking forward to more end-to-end projects, especially ones that involve metadata-driven pipelines and scheduled triggers for processing delta records daily. This project is great for someone just stepping into this field, looking forward to some complex projects as well.
Hi Ansh the video with detail explaining is very good. And you covered lot of things in detail 🎉🎉🎉.My side one request if possible can you create one video with scd-1 implementation with Adf , sql ,and dataflow combination end to end.
Hi, I just completed the full project, it is such a great video, it introduced me to most of the Azure resources as a beginner and you are a great teacher,so far the best tutorial among other similar RUclips videos, can you also do a video about triggers?
Make a project from end to end that we follow in industry * Bronze Layer -Raw Data folder Get the data from sftp and share point(use switch here base on source two copy data will work) - Processed Data Folder Create delta table upon above file(handle the excel file column heading properly) -Archived Folder Once table created all file will moved here *Silver -Cleaned the data and save in delta format -Create CDM layer(u can use this in Gold too) *Gold Layer - Create multiple kpi or dimension that will help power bi team for filter or hierarchy -aggregated table Use exception handling with raise error Video may be lengthy but compare with ur last two video as you made video that really teach compare to others Please make as no videos regarding industry level available in youtube
Hi Ansh, really appreciated for your efforts. I have seen many videos but yours is a simple and great video to learn for beginners. I am preparing for interview. if you can give info about challenges, blockages faced during project and root causes, Optimisation techniques used in ADF level and ADB level, strategy used for cost savings, and strategies used for secure data migration from onpremises/API to cloud and finally unit test strategies. Will be great if you help with above and any related links pls.
What are pre requisites for this video I have just just completed your video on data engineer fundamentals and loved it buddy and now I am keen to learn more about DE so thanks bro
Thank you for your positive feedback. Stay tuned bcz everything will be covered on this channel (Pls subscribe and share it with others). Happy Learning :)
Thank you for your positive feedback. In my next project video, I have created GOLD layer in Databricks and explained Delta Lake in detail. Stay tuned, will upload soon :)
Hi Ansh, I tried to link services https service testconnection was successful but when I want to link with datalakestorage it is getting failed. Apologies unable to attach screenshot
Hi Ansh, I have resolved this. The cause of the issue was that I didn't tick the hierarchical name after the enable tick option. test connection was successful
Good question, that shows you are really learning. So, while creating views, it's okay. But try to use dfs as much as you can. It's not mandatory but a good practice. Keep up the spirit and stay tuned for new video on every Sunday :)
Bro this just basic project we expect some dynamic approach like input data source you I have seen your project but in adls gen 2 location you just 1 folder and some files but this is static you have to mention when any customer will shared some extra files or folder that time how you handle this situation it's just static method Don't take me wrong I am just asking you..
Here's tip for you if you were unaware of it, always alter your skills and projects as per the JD. So, try to alter the bullet points highlighting these Azure tools. You can also use AI to alter it. Happy Learning :)
I am happy that you found my content insightful and want to pursue online course. Just subscribe this channel bcz lots of insightful content (related to Databricks) is on its way. Happy Learning :)
@@senthilonetwo I am happy to respond again. I'm happy that you like the way I teach. Stay tuned, everything will be covered in the upcoming videos on this channel. And a hint for you, a Databricks Masterclass video will be out soon :)
Hey, i have 12 years of experience as Software QA. I want to take transition into Data engineering. I will be mentioning in the interview that i have 3 years of experience in Azure Data Engineering out of 12 years. Can i explain this project in my interview? I mean is this project ideal for candidate who is into data engineering for 3-4 years ?
I am happy that you love my content. You will become an Azure Data Engineer by just following my content bcz on this channel, I will be putting everything. Stay tuned for new video on every Sunday :)
@@AnshLambaJSR Thank you so much. I have been trying to transition into Azure DE and been tired of looking for good resources. I have found your End-end video as the most simplest and best one i have seen on Azure End-End, and also the way you are encouraging when we gets errors is something motivating to keep doing it. Please make end-end vidoe for Azure Databricks, ADF, Synapse and Pyspark. I cannot wait for it. Thank you so much.
hey i got this error "Schema import failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'saazurelearn22'. FileSystem: 'bronze'. Path: 'products/products.csv'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: 'fcf61972-a01f-00c9-0c18-31c25d000000'. TimeStamp: 'Thu, 07 Nov 2024 13:27:03 GMT'. Operation returned an invalid status code 'NotFound'
Activity ID: 12bf639c-dcfa-42c5-875a-d96f0c43ccbb" while acessing azure data lake why ??
hey buddy trust me, you are doing just great i watched many videos on youtube almost all creators, but you are gem bro.... just loved the way you are teaching, thanks you brother...
Thank you so much for your lovely words. I am glad you loved my content. Stay tuned and share it with others :)
Usually, I don't comment on RUclips videos but the quality of this Azure end-to-end project video is so high that I couldn't resist commenting. This is way better than many fancy paid courses and udemy courses. Keep up the great work Ansh, best wishes!!
I am so happy that my video really helped you a lot. Please subscribe and share it with others. Lots of content is coming to your way every SUNDAY. Stay Tuned!
buddy, you're amazing! thank you so much for sharing your knowledge with us. i truly appreciate it. honestly, you're doing an incredible job!
This is real end to end azure data engineering project that I have ever seen on youtube, great work👍
thank you ! Please subscribe and share it with others.
This project is fantastic, and every step is explained in great detail-truly appreciate the effort! Looking forward to more end-to-end projects, especially ones that involve metadata-driven pipelines and scheduled triggers for processing delta records daily. This project is great for someone just stepping into this field, looking forward to some complex projects as well.
I am glad that you found my video helpful. Stay tuned for new video every week :)
This is indeed a good start and good coverage for beginners in the data Engg space. Great work!
Thank you for your kind words. I am glad that you liked it :)
just i want to start my interviews and i got this project before thanks man very helpfull
I am glad that my content is helping you. Stay tuned, a new amazing video is coming to you in the next hour :)
hi Ansh.
thanks for this incredible end to end project. love it. its a great hands-on data engineering on azure,
have ever seen... keep going bro...
I am glad that you found my content helpful. Check my latest video for more insights :)
Hi Ansh the video with detail explaining is very good. And you covered lot of things in detail 🎉🎉🎉.My side one request if possible can you create one video with scd-1 implementation with Adf , sql ,and dataflow combination end to end.
I am happy that you found my video useful. Sure, stay tuned for more! Happy Learning :)
Thanks a lot for this mate, great content built with lots of hard work and passion !! Keep more contents coming!!
I am so happy that my content helped you to learn a lot. Pls subscribe and share it with others :)
nice explain..doing great work..
Super bro . Congratulations on grand feedback 🎉👏 looking forward for more quality contents
Thank you for your positive feedback. I am glad that you liked my content. Stay tuned for more :)
Its very useful .I am trying to enter into Data Engineering Domain. The content is awesome. You nailed it bro.
Thank you so much. I am glad that my content helped you a lot. Please subscribe and share it with other data enthusiasts as well :)
Hi, I just completed the full project, it is such a great video, it introduced me to most of the Azure resources as a beginner and you are a great teacher,so far the best tutorial among other similar RUclips videos, can you also do a video about triggers?
Thank you so much for your kind words. I am glad that you loved my content. Sure, pls subscribe and stay tuned :)
Thanks Ansh.. Can you create one video on SCD types with Pyspark and with SQL?
Thank you for the positive feedback. Stay tuned and Happy Learning :)
Make a project from end to end that we follow in industry
* Bronze Layer
-Raw Data folder
Get the data from sftp and share point(use switch here base on source two copy data will work)
- Processed Data Folder
Create delta table upon above file(handle the excel file column heading properly)
-Archived Folder
Once table created all file will moved here
*Silver
-Cleaned the data and save in delta format
-Create CDM layer(u can use this in Gold too)
*Gold Layer
- Create multiple kpi or dimension that will help power bi team for filter or hierarchy
-aggregated table
Use exception handling with raise error
Video may be lengthy but compare with ur last two video as you made video that really teach compare to others
Please make as no videos regarding industry level available in youtube
Thank you for your feedback. Stay tuned to receive the insightful content. Will work on this too :)
@@AnshLambaJSR Bro use incremental loading too dont use auto loader rather use batch processing way
Awsome Brother ... Eagerly Waiting for More Real time Projects
I am glad that my content is help you. Stay tuned for more. Lots of content is lined up.
Expecting more project keep on update
Yes, many videos are lined up
Hi Ansh, really appreciated for your efforts. I have seen many videos but yours is a simple and great video to learn for beginners. I am preparing for interview. if you can give info about challenges, blockages faced during project and root causes, Optimisation techniques used in ADF level and ADB level, strategy used for cost savings, and strategies used for secure data migration from onpremises/API to cloud and finally unit test strategies. Will be great if you help with above and any related links pls.
Thank you for your positive feedback. I am really happy that you liked my content. Stay tuned for the above mentioned topics :)
Thank you for providing us with the valuable content.
You're welcome. Happy Learning!
Hey you are doing really good.hope you continue to make such tutorials
Thank you for your kind words. Yess, many tutorials are lined up and will be uploaded every week on Sunday. Stay tuned and subscribe :)
What are pre requisites for this video I have just just completed your video on data engineer fundamentals and loved it buddy and now I am keen to learn more about DE so thanks bro
Good content . will love to know if data is coming from multiple source how to implement lineage or implement meta data
Thank you for your positive feedback. Stay tuned bcz everything will be covered on this channel (Pls subscribe and share it with others). Happy Learning :)
Hi bro how are you..i have done my first part of pipeline dynamic runs succeed. Thanks lot bro
Yeahh.. I am glad that you learnt this and completed it.
I was told the same by my friend, next! next! next! ......... and done :D
haha! Please subscribe and stay tuned for more insightful content in the Data Engineering domain (Fun Sessions) !
Good content and loved your efforts. I have one doubt that is why synapse is used for gold layer? Can't we do it through databricks? If yes, then how?
Thank you for your positive feedback. In my next project video, I have created GOLD layer in Databricks and explained Delta Lake in detail. Stay tuned, will upload soon :)
@@AnshLambaJSR waiting for your next video ♥️
Can you make more videos like this
I am glad that you like my content. Yes, stay tuned! Lots of useful content is coming to your way every week on Sunday :)
Anyone please tell me i am learning ml algorithms and deep learning libraries like pytorch . Is the data engineering needed
You can always upskill yourself by learning Data Engineering as Data is growing rapidly :)
please create end to end FABRIC PROJECT
Thank you for the feedback. Will create soon :)
Finaly u started a youtube channel happy to see u here
Thank you. Please subscribe and share it with others
Hi Ansh, I tried to link services https service testconnection was successful but when I want to link with datalakestorage it is getting failed. Apologies unable to attach screenshot
Hi Ansh, I have resolved this. The cause of the issue was that I didn't tick the hierarchical name after the enable tick option. test connection was successful
@@sourabhkamble1766 I am glad you were able to learn this. Happy learning :)
HI,
while creating views you have given file name as blob right in earlier step you told to use dfs is that ok??
Good question, that shows you are really learning. So, while creating views, it's okay. But try to use dfs as much as you can. It's not mandatory but a good practice. Keep up the spirit and stay tuned for new video on every Sunday :)
Better than ramesh databricks course?
Just wanted to ask because i want to start some project for interviews??
hey, this is an end-to-end Azure Data Engineering Project. I hope you will learn a lot including real time scenarios. Happy Learning :)
Bro this just basic project we expect some dynamic approach like input data source you I have seen your project but in adls gen 2 location you just 1 folder and some files but this is static you have to mention when any customer will shared some extra files or folder that time how you handle this situation it's just static method
Don't take me wrong I am just asking you..
I am glad that you completed this project. In the future projects, you will see more dynamic approach. Stay tuned :)
How to write on fresher resume this project ... plz share the 3 to 4lines
You can alter your resume as per JD. Try using keywords such as Azure Data Factory, Azure Synapse, Databricks in your bullet points. :)
@@AnshLambaJSR tq 🫶
@@nsgamer...1635 always welcome :)
@@AnshLambaJSR what is the title of the project. plz reply me .
@@nsgamer...1635 Adventure Works End-To-End Azure DE project
Hi bro
Can you please share your resume where you including this project 🙏🏻🙏🏻
So that we know how to frame in resume please
Here's tip for you if you were unaware of it, always alter your skills and projects as per the JD. So, try to alter the bullet points highlighting these Azure tools. You can also use AI to alter it. Happy Learning :)
Dear, I am a beginner. Do I need prerequisites, or can I start there?
By the way, I have good knowledge of SQL, Python, NumPy, and Pandas.
It is beginner friendly. Everything is covered from scratch. Please subscribe and share it with others
Hi, May I know that you are taking Azure Databricks online course? If so I am interested doing it, Thanks.
I am happy that you found my content insightful and want to pursue online course. Just subscribe this channel bcz lots of insightful content (related to Databricks) is on its way. Happy Learning :)
@@AnshLambaJSR Have you read my question? your resposne looks like automated reply 😞😞😞
@@senthilonetwo I am happy to respond again. I'm happy that you like the way I teach. Stay tuned, everything will be covered in the upcoming videos on this channel. And a hint for you, a Databricks Masterclass video will be out soon :)
Hey, i have 12 years of experience as Software QA. I want to take transition into Data engineering. I will be mentioning in the interview that i have 3 years of experience in Azure Data Engineering out of 12 years. Can i explain this project in my interview? I mean is this project ideal for candidate who is into data engineering for 3-4 years ?
You can definitely explain this project and will help you ace your interviews. Stay tuned for more :)
Bro please change the talking accent, 🙏
Taken as feedback to channel growth
Feedback Taken!
Hi bro are you planning to take any course in online I want to become azure data engineer
I am happy that you love my content. You will become an Azure Data Engineer by just following my content bcz on this channel, I will be putting everything. Stay tuned for new video on every Sunday :)
@@AnshLambaJSRthanks bro and I will follow after getting a job I let you know the full credit for you only 🎉
@@sandhanamurali444 Hoping for your success. Happy Learning :)
subscribed
Thank you for supporting me :)
Hi bro. Do you take Azure Data Engineering course? I mean End-end course? I am ready to pay and join bro please
Thank you so much for the kind words. Stay tuned bcz lots of content is coming to your way to provide you lots of skills. Happy Learning.
@@AnshLambaJSR Thank you so much. I have been trying to transition into Azure DE and been tired of looking for good resources. I have found your End-end video as the most simplest and best one i have seen on Azure End-End, and also the way you are encouraging when we gets errors is something motivating to keep doing it. Please make end-end vidoe for Azure Databricks, ADF, Synapse and Pyspark. I cannot wait for it. Thank you so much.
@@connectwithanandsuresh I am happy that you love my content so much. Stay tuned for more and share it with others :)
For 4y experience can i add this project?
Definitely. Stay tuned to get more insightful content :)
Bro makes a Video for AWS End-To-End Data Engineering Project (From Scratch!)
I'm glad you liked this video. super soon :)
hey i got this error
"Schema import failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'saazurelearn22'. FileSystem: 'bronze'. Path: 'products/products.csv'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: 'fcf61972-a01f-00c9-0c18-31c25d000000'. TimeStamp: 'Thu, 07 Nov 2024 13:27:03 GMT'.
Operation returned an invalid status code 'NotFound'
Activity ID: 12bf639c-dcfa-42c5-875a-d96f0c43ccbb"
while acessing azure data lake why ??
Just make sure you're using right storage path. Try to focus on the steps explained in the video. Stay calm and try again