An End to End Azure Data Engineering Real Time Project Demo | Get Hired as an Azure Data Engineer
HTML-код
- Опубликовано: 22 май 2024
- #azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault
This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
The tools that are covered in this project are,
1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI
The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.
Follow the below link to import the database which is used in this project to the SSMS (I used the light weight version)
learn.microsoft.com/en-us/sql...
Chapters
TimeStamp:
0:00:00 - Project Overview
0:11:02 - Environment Setup
0:19:40 - Data Ingestion using ADF (Part 1)
0:41:31 - Data Ingestion using ADF (Part 2)
1:01:42 - Data Transformation using Databricks (Part 1)
1:16:22 - Data Transformation using Databricks (Part 2)
1:36:26 - Data Transformation using Databricks (Part 3)
1:51:58 - Data Loading using Azure Synapse Analytics
2:15:34 - Data Reporting using Microsoft Power BI
2:30:35 - Creating Security Group using Azure Active Directory (AAD)
2:36:44 - End to End Pipeline Testing
- - - Book a Private One on One Meeting with me (1 Hour) - - -
www.buymeacoffee.com/mrktalks...
- - - Express your encouragement by brewing up a cup of support for me - - -
www.buymeacoffee.com/mrktalks...
- - - Other useful playlist: - - -
Azure Data Factory Playlist: • Azure Data Factory Tut...
Azure General Topics Playlist: • Azure Beginner Tutorials
Microsoft Fabric Playlist: • Microsoft Fabric Tutor...
Azure Databricks Playlist: • Azure Databricks Tutor...
Azure End to End Project Playlist: • End to End Azure Data ...
Databricks CICD Playlist: • CI/CD (Continuous Inte...
End to End Azure Data Engineering Project: • An End to End Azure Da...
- - - Let’s Connect: - - -
Email: mrktalkstech@gmail.com
Instagram: mrk_talkstech
- - - Tools & Equipment (Gears I use): - - -
Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!
DJI Mic: amzn.to/3sNpDv8
Dell XPS 13 Plus 13.4" 3.5K : amzn.to/45KqH1c
Rode VideoMicro Vlogger Kit: amzn.to/3sVFW8Y
DJI Osmos Action 3: amzn.to/44KYV3x
DJI Mini 3 PRO: amzn.to/3PwRwAr
- - - About me: - - -
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst
3. Data Scientist
Can't wait to help people with my videos.
- - - Support me: - - -
Please Subscribe: / @mr.ktalkstech 
Наука
I hope you find this video useful. If you have learned something or found it helpful, please consider giving it a like.
What topic should I cover in the next end-to-end project video? Please let me know your thoughts in the comments.
Follow my LinkedIn profile to get regular updates:
www.linkedin.com/in/mrk-talkstech/
Thank you!😊😊😊
Please do project on autoloader incremental pipeline with dimensions, facts, quality checks, phase.
You could do a video migrating on premises to synapse, for example , loading facts and dims using serveless and delta tables, using databricks …congrats by your videos, your video is better than Microsoft official courses
Please do project for data warehouse .
instead of all tables how to choose custom tables, and push only those tables on prem sql-server to ADLS.
@hello sir what will be cluster configuration for real time projects
This is the video I have been searching for a long time to kick off my databricks journey, now Mr. K you offered it for public free, thank you very much. :) Please keep it up.
Excellent and well described end to end data pipeline. Waiting for more videos on real time and incremental batch processing using ADF and databricks. Thanks
Please come up with more videos.Thanks a lot for your effort
Thanks for this project, expecting such more end to end projects.
Hello, thank you for an excellent explanation on the basic end to end project. Looking forward to learn more from you. Keep posting the great stuff often.
An amazon tutorial presented with clarity and professionalism. Thank you so much for the great work!
This channel is definitely my go to place for End to End Projects . Absolutely love your content sir . Keep it coming . Thank you 😊
Thank you so much :)
Crispy clear, no-nonsense, dense, structured. Definitely subscribed.
Thank you so much :)
Mr. K, you are the gem man. Your way of explaining the things superb. Thanks a lot for this free content.
Thank you so much :)
Great video! Your explanation of End-to-end Azure Data Engineering was clear, simple and easy to follow. I appreciate your expertise in breaking it down for us. God bless!!
Thank you so much :)
Thank you MrK, I’m a project manager with not so tech background and I look after these kinds of projects. Your explanation gave me perfect idea of understanding the flow functionally and made me aware of Data engineering projects end to end, thank you!
Crystal clear explanation & Amazing content Mr.K, Looking forward for more Hands-on projects.
Thank you for this contribution.
Thank you so much :)
Please bring some more End-to-End projects on Azure
First time, I've encountered a comprehensive and practical explanation from start to finish. Thank you very much for all your efforts.
Thank you so much :)
What a powerful 1st 10 minutes... Looking forward to following along! I am just a SQL DB to Power Bi, but want to push the company forward and this is just the knowledge I need to begin :)
Thank you soo much :)
You sir, have earned my respect. Excellent communication style, clear, and concise. Very well done. 👍🏼
Thank you so much :)
"Great video! Your explanation of End-to-end Azure Data Engineering was clear and easy to follow. I appreciate your expertise in breaking it down for us.
Thank you so much :)
Great work ! It is the best tutorial so far about Azure data engineering on RUclips.
Thank you so much :)
Your presentation and explanation were excellent, and Microsoft owes you appreciation for it!!
Thank you so much :)
excellent video to start the journey with azure..thanks a lot
It's my first video from your channel and I loved it. I liked the clear and concise explanation, along with utilizing various services in creating a truly end to end Data Engineering Project. I was able to successfully implement the pipelines after learning from your video. Thank you!
Thank you so much :) Glad you liked it :)
Could you pls share project name n roles n responsibilities
Very well crafted, the best possible way to create an end to end project for someone who is a beginner to Azure.
A very good job, Keep going buddy🎉
Thank you so much :)
This is amazing, end to end, thorough and clear concept and example detail, love it
Thank you so much :)
This is great content! Thank you so much for sharing. Looking forward to watch the next end to end project videos
Thank you so much :)
Awesome tutorial....Thanks for making it so very easy to understand. Please make more videos.
Thank you very much for the efforts you put in to train us. Keep doing the good work and create more end to end projects like this with different tech stacks :)
Thank you so much :)
The BEST tutorial about data engineering that I have Found. I'm a data analyst, but in my company I'm having to do a LOT of data engineering job and I was totally lost till I found this tutorial. THANK you for helping me.
Bro same condition with me as well
But does the company give us data engineer experience and does the data analyst experience worth it for data engineer
Awesome project and really well explained. I learned a lot. I obviously built the same as the video progressed, although I used a more comprehensive database with multiple schemas. That forced me to learn how to create the pipelines slightly different. Really enjoyed every minute of the video. I like the end-to-end project approaches, you learn so much more and its closer to a real world data flow.
Thank you so much :)
wow, what an amazing demonstration of all the cloud products with hands on lab!!
Thanks
Thank you so much :)
Clear explanation with ADF BUT one needs to build more real time projects as we as with scenarios bro
All ways welcome brother...
Thank you for providing the detailed overview of ETL using Azure. This session was very helpful.
Thank you :)
This is the best end-to-end project I have ever seen. I was able to set up a whole data engineering solution and personally implement it in my company based on this video. Please add more end-to-end projects when you get time. Amazing tutor and amazing channel.
Thank you soo much :)
Hey, can you assist me with something? I am stuck badly on something
This is awesome, Mr. K. Absolutely fantastic.
Thank you so much :)
Fantastic Video.. Keep up the good work .
oh man. you are the man. everything was easy to understand . keep it going my man. you definitely earned a subscriber.
Thank you so much :)
A senior colleague recommended this video to me and I must say I am beyond grateful. Thanks for compiling such an amazing content. A Game Changer!!!
Thank you so much :)
I came here in search for Copper But I found Gold You are an amazing teacher Thank you ♦🙌
Thank you, was stuck at key vault-linked service section for sometime . your video helped me clear it.
Thank you so much :)
Thank you so much Mr. K 🤩
just 1 word...Superbbb
Excellent Video and Thanks a lot for your efforts for making this video .
very useful content, thanks for sharing!
Thanks a lot for the clear explanation!
Very helpful. Appreciated your effort.
Greatness! Thanks very much I have learnt alot
Amazing course with very clear explanation! just completed the whole project.
Thank you so much :)
Thanks a lot for this end to end project video. I got fair understanding and motivation about the Azure data engineer role. This video will help me to speedup my learning curve.
Thank you so much :)
This is a hidden Gem for sure. Will make sure to go through all your videos.
Thank you so much :)
This video was really helpful thanks for bringing this up 😊
Hope to see more end to end projects
Thank you so much :)
"Great video! Nice explanation of End-to-end Azure Data Engineering was clear and easy to follow. I must appreciate your expertise in breaking it down for us.
Thank you so much :)
Thank you Mr K Talks Tech !! .. It was really helpful for my career in Data engineering using Azure,, I will share this with my friends and keep this as a favorite for future reference
Thank you so much :)
One more end to end project, with different scenarios would be great
Very nice, very useful, thank you.
I wish I would have found this earlier, all these i have been finding bits and pieces. This is really end-to-end solution one should go through. I wish you should get as many views as possible. Really mind-blowing one single video to learn much. Stay blessed.
Thank you so much :)
Excellent video with great explanations - thanks so much!
Thank you so much :)
Very good presentation and explanation..
Awesome content very clear explanation sir. You are a great teacher! Lucky to have found this channel.
Thank you so much :)
I wouldn't be surprised if this channel hits a million subscribers soon !!! Excellent explanations and extra ordinary presentations 👏👏. WIll continue to watch every videos of this channel...love the structured way of going with the data related topics 👌
Thank you so much for the support and the biggest compliment :)
your explanation skills are amazing man. that's so hard to find. you dumb it down make things very accessible
Thank you so much :)
Just one word...Masterpiece.👏👏👏👏
Thank you so much :)
Thank you for the wonderful presentation in data engineering, "Kudos to the Mr.K Talks channel for breaking down complex data engineering concepts! Feeling empowered to tackle real-time projects with confidence it has helped a lot in understanding the real time project building, Thanks for the clarity and practical takeaways.
Thank you so much :)
amazing work Mr.K , keep it up !!
Thank you so much :)
Very helpful. Thank you
hi, great tutorial and indeed good learning for starters as me. Can you also please make end to end azure data engineering real time project with continuous data stream & readily available big data (so that we can readily download from your link). It would be of great help for us.
Very clear explanations along each step. Thank you very much. Would you do a video on Change Data Capture (CDC) and SCD Type with numerical examples. Thank You.
Great one i have been on vacation and i find this one. :)
This is the most action packed azure project i have seen till date. The instructor is really very knowledgeable. This project itself is far more informative than most of the paid course on ADF.
Thank you so much :)
Explanation was amazing 🎉
Excellent video.. all topics are explained in detail and covered all topics.Thanks for the video.
Thank you so much :)
Amazing video with very clear explanation. Thank you.
Thank you so much :)
Thanks Mr. K.. I finished creating the pipeline by following your procedure.. though I encountered some problems with setting up my on prem SQL database. Everything works now without any problems. I've learned a lot.. Thanks for sharing your knowledge. All the best.
Thank you so much :)
what error did u face bro
can u guide me furture and help me in this project ?
Thank you Mr. K.. 😀
the videos thati have gone through both from Udemy and RUclips, let me share that this content is far better than any of the videos i have gone through. He explains in deep how and why. Loving it
Thank you so much for the huge compliment :)
Thanks you so much Mr.k . it has more knowledge. Please make more videos .
Sure :)
It is a great effort that you have made in supporting those who cannot afford to spend money on courses. This channel may become successful in the coming days. The content you provide here is not taught in courses themselves. Please be motivated to continue teaching in this way. We are here to support anyway whatever possibility we have in our hands.
Thank you so much for the support :) Sure, will continue making the videos :)
Very very useful. Thanks for such an elaborative explanation. Kudos !!
Thank you so much :)
Coolest..Azure data engineering project i have seen till now..Great stuff
Thank you soo much :)
@@mr.ktalkstech where we can find dataset?
Excellent video .. i was keep on watching this channel to learn real time scenarios thank you so much my Azure Data engineer Guru
Thank you so much for such big compliments :)
@@mr.ktalkstech always welcome Guru..
I wants to switch to azure Data engineer..
Could you pls connect with me one-to-one once bro
The video was excellent, featuring clear explanations and valuable content.
Thank you so much :)
Thanks for making this video
It gives me full knowledge of the project as a beginner
Thank you so much :)
this video is very helpful, good hands on experience on azure services , the way you explained step by step process is very interesting, great content .
Thank you so much :)
Very nice Mr K ❤
Highest effort you have given. Kindly share the all code you have used in the video will be helpful. Thank you
Big thank you MrK. Much appreciated your detailed guidance. Cheers bhai ❤
Thank you so much :)
You're amazing. Thanks for the video bro. Very informative, such insight.
Thank you so much :)
speachless effort
Sir, what a in-dept explanation end to end project. I am working as Power Bi developer just begin to learn Azure Data engineering. You explained very well I got confidence that I can do it. Thanks a lot once again👍
Thank you so much :)
Thanks friend, you're really a grandmaster... this video is helping me a lot!!!
Thank you so much :)
Awesome Content Keep it up
Very useful ! Thanks for that ! keep this up
Thank you soo much :)
Amazing work. Appreciate the help. just one question , Can we use azure data brick instead Synapse to avoid vendor lock in ?
Please make more end to end real time project like this using azure. The video is so much informative and have learnt so many things.
Thank you so much, Sure :)
I have watched the entire video like a movie without any distraction and never slept in between. I have seen many videos to learn Azure but not satisfied. But you have explained excellently with in depth detail and highlighted the points wherever is required without missing any single part. Your dedication on creating such a wonderful tutorial even during late night Sundays is much appreciated. This end to end project gives full confidence to work as a Data Engineer in the Azure portal. Hats off to you Kishore. Semma mass!
Subscribed, liked and shared. Thanks a lot brother!
Thank you so much for the huge compliment :)
Woah !! Great content sir !! Thank you
Thank you so much :)
Thankyou for the video,very helpful🙂
Thank you so much :)
Wonderful presentation and explanation. It helps a lot to the beginner.
Thank you so much :)
Top Notch Video Mr Kishor Kumar sir..... can you make more videos like these, it will surely help the community to clear the concepts....easily with your videos....Again Thank you for this amazing videos...🔥🔥🔥🔥🔥
Thank you so much :)
Woww!! What a clear, knowledge-filled video! Thank you so much!!
One when to use what question between Synapse Analytics and Data Factory if both offer creating pipelines if want to use Databricks? Thank you in advance!
Crystal clear bro ... thank you
Thank you so much :)