- Видео 81
- Просмотров 1 259 107
Mr. K Talks Tech
Индия
Добавлен 27 дек 2022
Hello Everyone,
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst and
3. Data Scientist
Can't wait to help people with my videos. Stay tuned!
For contact:
Email: mrktalkstech@gmail.com
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst and
3. Data Scientist
Can't wait to help people with my videos. Stay tuned!
For contact:
Email: mrktalkstech@gmail.com
Part 2- End to End Realtime Streaming Azure Data Engineering Project (Environment Setup)
Welcome back to the next part of our exciting end-to-end real-time streaming Data Engineering project! In this video, we’ll walk through the entire environment setup process, covering all the essential resources we’ll use to build a real-time weather reporting system using Azure services and Microsoft Fabric.
TimeStamp:
0:00:00 - Setting up the Weather API
0:03:58 - Creating Resource Group
0:07:09 - Creating Azure Databricks
0:12:59 - Creating Azure Function APP
0:20:16 - Creating Azure Event HUB
0:25:24 - Creating Azure Key Vault, RBAC and Creating Secrets
0:36:08 - Creating Microsoft Fabric
#DataEngineering #AzureDatabricks #AzureFunctions #EventHub #MicrosoftFabric #RealTimeData #StreamingData ...
TimeStamp:
0:00:00 - Setting up the Weather API
0:03:58 - Creating Resource Group
0:07:09 - Creating Azure Databricks
0:12:59 - Creating Azure Function APP
0:20:16 - Creating Azure Event HUB
0:25:24 - Creating Azure Key Vault, RBAC and Creating Secrets
0:36:08 - Creating Microsoft Fabric
#DataEngineering #AzureDatabricks #AzureFunctions #EventHub #MicrosoftFabric #RealTimeData #StreamingData ...
Просмотров: 1 719
Видео
End to End Realtime Streaming Azure Data Engineering Project (Part -1) | Complete Guide with Demo
Просмотров 3,8 тыс.21 день назад
Welcome to an exciting session where we'll build a real-time streaming Data Engineering project using Azure services! This is one of the best ways to get hands-on experience with real-time streaming while understanding architectural decision-making for cost and performance optimization. In this project, we’ll: - Set up a real-time weather report system using Weather API - Implement data ingesti...
6 Proven Steps to Land a Data Engineer Job with ZERO Experience (Even as a Fresher!)
Просмотров 4,4 тыс.Месяц назад
#dataengineering #azuredataengineering #aws #gcp #jobs In this video, I have explained 6 important steps that will help you to get hired as a Data Engineer. This is probably the most asked question to me, and that's the reason I decided to make a video on this topic. I hope this is helpful. Please let me know if you have any questions in the comment section. - - - Book a Private One on One Meet...
Must-Know PySpark Interview Question for Data Engineers - Live Demo & Tips!
Просмотров 3 тыс.Месяц назад
#ApacheSpark #DataEngineering #AzureDataEngineer #SparkSQL #DataTransformation #DataFrame #InterviewQuestion #BigData #AzureDatabricks #PySpark #DataAnalysis #DataScience #SQLQuery #Optimization #Efficiency #Tutorial In this video, we'll dive into a popular PySpark interview question often asked by financial and banking companies-calculating the running total for grouped data. We'll explore the...
What is Apache Spark? Learn Apache Spark in 15 Minutes
Просмотров 4,9 тыс.2 месяца назад
#apachespark #databricks #sparkteam #dataengineering #pyspark #architecture In this video, I have covered the most important topic of Data Engineer which is "Apache Spark". Especially, I have talked about the complete end to end Architecture of Spark Spark covering all the individual components as below, 1. Driver Programme 2. Worker Node 3. Cluster Manager 4. Spark Context or Spark Session 5. ...
Data Engineer's Biggest Nightmare: Solving Merge Conflicts Step-by-Step Guide with Demo
Просмотров 1,9 тыс.3 месяца назад
#AzureDevOps #Git #DataEngineering #InterviewQuestions #DevOpsTutorial #AzureDatabricks #GitRevert #BranchRecovery #CodingBestPractices #TechInterviewTips #MergeConflicts 🔍 What We'll Cover: In this video, we dive into a crucial topic for every Data Engineer: Merge Conflicts. Merge conflicts are one of the most common issues encountered when working collaboratively with Git repositories. Unders...
Untold secrets of Data Engineering Project !!! Reality and Facts in 15 minutes
Просмотров 8 тыс.3 месяца назад
#DataEngineering #ProjectCreation #RealTimeDataEngineering #DataEngineeringTutorial #DataEngineeringProject #PreImplementationPhase #ImplementationPhase #PostImplementationPhase #DataEngineers #SolutionArchitects #DevOpsEngineers #InfrastructureEngineers #ProjectManagers #ModernDataPlatform #Terraform #CICDPipelines #DataPlatformMaintenance #DataEngineeringCareer #DataEngineeringInterviewTips #...
Don't fail to answer these questions in your next Interview- Data Engineering Interview Question
Просмотров 3,1 тыс.4 месяца назад
#AzureDevOps #Git #DataEngineering #InterviewQuestions #DevOpsTutorial #AzureDatabricks #GitRevert #BranchRecovery #CodingBestPractices #TechInterviewTips 🔍 What We'll Cover: In this video, we tackle two essential Azure DevOps and Git-related interview questions for data engineers. The first question addresses how to revert changes mistakenly committed to the main branch. Using a practical exam...
Tips and Tricks- Delta Lake Table in Apache Spark - Azure Data Engineering Interview Question
Просмотров 4 тыс.4 месяца назад
#PySpark #DeltaTable #AzureDatabricks #BigData #DataEngineering #DataLake #InterviewQuestions #TechTutorial #DataAnalytics #DataTransformation #SchemaEvolution #ACIDProperties #DataGovernance #TimeTravel #DataPipelines #Azure #Databricks 🔍 What We'll Cover: Delta Table Overview: Understand why Delta Tables have become a popular choice for big data analytics in cloud-based platforms. Code Walkth...
Can you solve this simple Spark SQL Interview Question? | Azure Data Engineering Tutorials
Просмотров 1,9 тыс.4 месяца назад
#SQL #SPARKSQL #Databricks #SQLInterview #SQLTutorial #DataScience #DataEngineering #Programming #Coding #TechInterviews In this video, we explore a SPARK SQL interview question that may seem simple but can be quite tricky. This question has stumped many candidates recently, which is why we're diving deep into it. If you find this video helpful, please like, share, and subscribe for more tutori...
15 Minutes- Libraries in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials
Просмотров 2,8 тыс.4 месяца назад
#AzureDatabricks #DatabricksLibraries #DataTransformation #MachineLearning #DataScience #PythonLibraries #ExternalLibraries #NotebookScopedLibraries #ClusterScopedLibraries #FastText #NumPy #DatabricksTutorial #DataEngineering #CloudComputing #BigData #AI #InterviewPrep #TechTutorials #DataAnalytics #Programming #Development #Coding #Azure #CloudData Welcome to our latest tutorial on managing l...
Azure Synapse Analytics- Interview Questions | Serverless SQL Pool VS Dedicated SQL Pool with Demo
Просмотров 4,4 тыс.5 месяцев назад
#AzureSynapse #SQLPools #InterviewPrep #DataAnalytics #CloudComputing #Azure #SynapseAnalytics #Serverless #DedicatedSQL #DataManagement #Database #DataWarehousing #BigData #TechExplainer #TechTutorial #TechEducation #TechInsights #DataScience In this video, we're diving deep into the world of Azure Synapse Analytics, focusing specifically on Serverless SQL Pools vs Dedicated SQL Pools. If you'...
Most popular Data Warehousing Interview Question- Azure Data Engineering Interview Question Tutorial
Просмотров 3,1 тыс.6 месяцев назад
#ApacheSpark #DataEngineering #AzureDataEngineer #SparkSQL #DataTransformation #DataFrame #InterviewQuestion #BigData #AzureDatabricks #PySpark #DataAnalysis #DataScience #SQLQuery #Optimization #Efficiency #Tutorial Welcome to our latest video where we delve into an essential interview question for Azure Data Engineers focusing on Type1 vs Type 2 in Data warehousing. As Data Engineers, it's re...
90% of Data Engineers doesn't know this - PySpark Azure Data Engineering Interview Question Tutorial
Просмотров 4,8 тыс.6 месяцев назад
#ApacheSpark #DataEngineering #AzureDataEngineer #SparkSQL #DataTransformation #DataFrame #InterviewQuestion #BigData #AzureDatabricks #PySpark #DataAnalysis #DataScience #SQLQuery #Optimization #Efficiency #Tutorial Welcome to our latest video where we delve into an essential interview question for Azure Data Engineers focusing on Apache Spark. Specifically, we tackle data transformation witho...
Build Your End-to-End Project in Just 3 Hours: Master Azure Data Engineering | Bing Data Analytics
Просмотров 19 тыс.6 месяцев назад
#AzureDataEngineering #MicrosoftFabric #DataFactory #SynapseAnalytics #DataIngestion #DataTransformation #DataScience #DataAnalytics In this video, I'll show you how to build a Bing News Data Analytics platform! We'll do a big project in Azure Data Engineering using Microsoft Fabric. We'll pull raw data from Bing API, transform the raw data to clean data with Synapse Data Engineering, analyze s...
Tips and Tricks- Azure Data Engineering Interview Questions | Managed Identity vs Service Principal
Просмотров 11 тыс.6 месяцев назад
Tips and Tricks- Azure Data Engineering Interview Questions | Managed Identity vs Service Principal
(Final) Part 9: Setting up Alerts using Data Activator and End to End Pipeline Testing | Azure Demo
Просмотров 2,7 тыс.7 месяцев назад
(Final) Part 9: Setting up Alerts using Data Activator and End to End Pipeline Testing | Azure Demo
Building Pipelines using Data Factory(Part-8)-End to End Azure Data Engineering Project using Fabric
Просмотров 3 тыс.7 месяцев назад
Building Pipelines using Data Factory(Part-8)-End to End Azure Data Engineering Project using Fabric
Building Reports using Power BI (Part-7) - End to End Azure Data Engineering Project using Fabric
Просмотров 3,2 тыс.7 месяцев назад
Building Reports using Power BI (Part-7) - End to End Azure Data Engineering Project using Fabric
Sentiment Analysis using Synapse ML(Part-6)- End to End Azure Data Engineering Project using Fabric
Просмотров 3,2 тыс.8 месяцев назад
Sentiment Analysis using Synapse ML(Part-6)- End to End Azure Data Engineering Project using Fabric
Incremental Load- Type 1(Part-5) - End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 4,7 тыс.8 месяцев назад
Incremental Load- Type 1(Part-5) - End to End Azure Data Engineering Project using Microsoft Fabric
Data Transformation (Part-4) - End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 6 тыс.8 месяцев назад
Data Transformation (Part-4) - End to End Azure Data Engineering Project using Microsoft Fabric
Data Ingestion (Part-3) - End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 7 тыс.8 месяцев назад
Data Ingestion (Part-3) - End to End Azure Data Engineering Project using Microsoft Fabric
(Part-2) Environment Setup- End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 9 тыс.8 месяцев назад
(Part-2) Environment Setup- End to End Azure Data Engineering Project using Microsoft Fabric
Bing News Data Analytics- End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 23 тыс.9 месяцев назад
Bing News Data Analytics- End to End Azure Data Engineering Project using Microsoft Fabric
A 6-Month Complete Guide to Become a Data Engineer in 2024 😎 - Azure Data Engineer Roadmap 2024
Просмотров 26 тыс.10 месяцев назад
A 6-Month Complete Guide to Become a Data Engineer in 2024 😎 - Azure Data Engineer Roadmap 2024
Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials
Просмотров 8 тыс.11 месяцев назад
Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials
Top 5 Notebook Features You Can't Miss in Azure Databricks 🚀| Databricks and Azure DevOps Tutorials
Просмотров 3,5 тыс.11 месяцев назад
Top 5 Notebook Features You Can't Miss in Azure Databricks 🚀| Databricks and Azure DevOps Tutorials
15 Minutes- Spark Clusters in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials
Просмотров 19 тыс.Год назад
15 Minutes- Spark Clusters in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials
Master Azure Databricks CI/CD in 2 Hours with Azure DevOps | Full End-to-End CI/CD Project in Azure
Просмотров 44 тыс.Год назад
Master Azure Databricks CI/CD in 2 Hours with Azure DevOps | Full End-to-End CI/CD Project in Azure
Upload remaining part
The third part please
Thank you for the video. May I ask if you runned this project with Pay-as-you-go or Free subscription?
"PERMISSION_DENIED: Invalid permissions on the specified KeyVault" is the error i came here for and thanks very much. It really helped
Please drop the next video 😢😅, anticipation want to kill me.
Brilliant Sir! it i something COMPLETE, really end to end. I am really enjoying watching this video.
Can you please share the YML scrips. Or please share Github link for that.
How to od this with workflows?
waiting for next video, Can you please upload Microsoft fabric videos...
Superb explanation. Can you please make a one how to handle real time streaming data using stream analytics with end to end project..
great sir , Happy Diwali 🙂
Mr.K can you please make a separate video for CICD integration in Fabric? specially for the free trial account
It seems like nowadays it is not possible to create compute resources in databricks with free subscription. You need to have pay as you go subscription. Is it true or it's only happening for me.
I like your statement "As you can see that " you have completed the final part :) Thanks a lot
U r explaining like Robot...Everything is perfect.... Can u make this videos in Hindi
bro when is part 3 rolling out
Thank you very much, i also added the category filter so that i can only see the news like sports, politics etc and also check the sentiment on those verticles
Awesome Explanation
We can use df.groupby().agg() right?
thanks!
Great one, waiting for the next part in the series.
Lots of learning from this session, its really helpful. Thank you so much Mr. K... your teaching style is unique and very clear in concept with simple business cases.👌
crystal clear explanation :) Thanks
Thank you so much for your videos. Excellent...!
Thank you. Do you have a community ? I wanna join please.
hey man, may be worth a shot checking out LakeSail's PySail built on rust. supposedly 4x faster with 90% less hardware a cost according to their latest benchmarks. and can migrate existing python code. might be cool to make a vid on! love ur content!
You explained well about all concepts of Azure data factory, could please make more playlist for beginners from basic to advanced like SQL, pyspark, Azure storage gen2, required python concepts for data engineering, etc. So that we learn from basic to advance
Big Shoutout to you brother, you just made the conceptual understanding easier ,keep going
this video is savior for new aspirants
This is the cleanest explanation I have ever come across on azure.
you have already written the code and explaining it makes difficult to understand
This tutorial is actually not for beginners Upload it properly how to install platforms to create those setup & all etc.
How can we use Azure databricks in this project if we have the Trial version Azure Account(cluster running is prohibited in trial) ? Is there any other way? Can use Fabric for spark and azure functions?
Pls do the video about SCD Type1 and SCD Type2 anna
great video, thank u for your efforts, I can't wait for the next section,
please always include code use in your video thanks
I appreciate your explanation; it has clarified the topic for me. Thank you. 🙏🏼 However, I have one question: if the CSV files are split into two, how will one worker determine if there are any duplications with another worker work?
Could you please tell us in how many parts you are going to finish this project ? Thanks
Great help , thanks ✅
Amazing content quality & explaination. Its the best data engineering project explained. Keep it up Sir.
Thank you!
Thank you anna you are my GURU❤️
Thank you🙏
Thank you!
Mr.k ,Thanks for the project and expecting more end to end projects .
Thank you for sharing valuable information and i really appreciate your efforts.
Upload tamil
This is great and very detailed!!!
Thank you Mr KT
Your videos are infinitely helpful! Thank you so much for all the hard work you put into them. Very useful!