Mr. K Talks Tech
Mr. K Talks Tech
  • Видео 81
  • Просмотров 1 259 107
Part 2- End to End Realtime Streaming Azure Data Engineering Project (Environment Setup)
Welcome back to the next part of our exciting end-to-end real-time streaming Data Engineering project! In this video, we’ll walk through the entire environment setup process, covering all the essential resources we’ll use to build a real-time weather reporting system using Azure services and Microsoft Fabric.
TimeStamp:
0:00:00 - Setting up the Weather API
0:03:58 - Creating Resource Group
0:07:09 - Creating Azure Databricks
0:12:59 - Creating Azure Function APP
0:20:16 - Creating Azure Event HUB
0:25:24 - Creating Azure Key Vault, RBAC and Creating Secrets
0:36:08 - Creating Microsoft Fabric
#DataEngineering #AzureDatabricks #AzureFunctions #EventHub #MicrosoftFabric #RealTimeData #StreamingData ...
Просмотров: 1 719

Видео

End to End Realtime Streaming Azure Data Engineering Project (Part -1) | Complete Guide with Demo
Просмотров 3,8 тыс.21 день назад
Welcome to an exciting session where we'll build a real-time streaming Data Engineering project using Azure services! This is one of the best ways to get hands-on experience with real-time streaming while understanding architectural decision-making for cost and performance optimization. In this project, we’ll: - Set up a real-time weather report system using Weather API - Implement data ingesti...
6 Proven Steps to Land a Data Engineer Job with ZERO Experience (Even as a Fresher!)
Просмотров 4,4 тыс.Месяц назад
#dataengineering #azuredataengineering #aws #gcp #jobs In this video, I have explained 6 important steps that will help you to get hired as a Data Engineer. This is probably the most asked question to me, and that's the reason I decided to make a video on this topic. I hope this is helpful. Please let me know if you have any questions in the comment section. - - - Book a Private One on One Meet...
Must-Know PySpark Interview Question for Data Engineers - Live Demo & Tips!
Просмотров 3 тыс.Месяц назад
#ApacheSpark #DataEngineering #AzureDataEngineer #SparkSQL #DataTransformation #DataFrame #InterviewQuestion #BigData #AzureDatabricks #PySpark #DataAnalysis #DataScience #SQLQuery #Optimization #Efficiency #Tutorial In this video, we'll dive into a popular PySpark interview question often asked by financial and banking companies-calculating the running total for grouped data. We'll explore the...
What is Apache Spark? Learn Apache Spark in 15 Minutes
Просмотров 4,9 тыс.2 месяца назад
#apachespark #databricks #sparkteam #dataengineering #pyspark #architecture In this video, I have covered the most important topic of Data Engineer which is "Apache Spark". Especially, I have talked about the complete end to end Architecture of Spark Spark covering all the individual components as below, 1. Driver Programme 2. Worker Node 3. Cluster Manager 4. Spark Context or Spark Session 5. ...
Data Engineer's Biggest Nightmare: Solving Merge Conflicts Step-by-Step Guide with Demo
Просмотров 1,9 тыс.3 месяца назад
#AzureDevOps #Git #DataEngineering #InterviewQuestions #DevOpsTutorial #AzureDatabricks #GitRevert #BranchRecovery #CodingBestPractices #TechInterviewTips #MergeConflicts 🔍 What We'll Cover: In this video, we dive into a crucial topic for every Data Engineer: Merge Conflicts. Merge conflicts are one of the most common issues encountered when working collaboratively with Git repositories. Unders...
Untold secrets of Data Engineering Project !!! Reality and Facts in 15 minutes
Просмотров 8 тыс.3 месяца назад
#DataEngineering #ProjectCreation #RealTimeDataEngineering #DataEngineeringTutorial #DataEngineeringProject #PreImplementationPhase #ImplementationPhase #PostImplementationPhase #DataEngineers #SolutionArchitects #DevOpsEngineers #InfrastructureEngineers #ProjectManagers #ModernDataPlatform #Terraform #CICDPipelines #DataPlatformMaintenance #DataEngineeringCareer #DataEngineeringInterviewTips #...
Don't fail to answer these questions in your next Interview- Data Engineering Interview Question
Просмотров 3,1 тыс.4 месяца назад
#AzureDevOps #Git #DataEngineering #InterviewQuestions #DevOpsTutorial #AzureDatabricks #GitRevert #BranchRecovery #CodingBestPractices #TechInterviewTips 🔍 What We'll Cover: In this video, we tackle two essential Azure DevOps and Git-related interview questions for data engineers. The first question addresses how to revert changes mistakenly committed to the main branch. Using a practical exam...
Tips and Tricks- Delta Lake Table in Apache Spark - Azure Data Engineering Interview Question
Просмотров 4 тыс.4 месяца назад
#PySpark #DeltaTable #AzureDatabricks #BigData #DataEngineering #DataLake #InterviewQuestions #TechTutorial #DataAnalytics #DataTransformation #SchemaEvolution #ACIDProperties #DataGovernance #TimeTravel #DataPipelines #Azure #Databricks 🔍 What We'll Cover: Delta Table Overview: Understand why Delta Tables have become a popular choice for big data analytics in cloud-based platforms. Code Walkth...
Can you solve this simple Spark SQL Interview Question? | Azure Data Engineering Tutorials
Просмотров 1,9 тыс.4 месяца назад
#SQL #SPARKSQL #Databricks #SQLInterview #SQLTutorial #DataScience #DataEngineering #Programming #Coding #TechInterviews In this video, we explore a SPARK SQL interview question that may seem simple but can be quite tricky. This question has stumped many candidates recently, which is why we're diving deep into it. If you find this video helpful, please like, share, and subscribe for more tutori...
15 Minutes- Libraries in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials
Просмотров 2,8 тыс.4 месяца назад
#AzureDatabricks #DatabricksLibraries #DataTransformation #MachineLearning #DataScience #PythonLibraries #ExternalLibraries #NotebookScopedLibraries #ClusterScopedLibraries #FastText #NumPy #DatabricksTutorial #DataEngineering #CloudComputing #BigData #AI #InterviewPrep #TechTutorials #DataAnalytics #Programming #Development #Coding #Azure #CloudData Welcome to our latest tutorial on managing l...
Azure Synapse Analytics- Interview Questions | Serverless SQL Pool VS Dedicated SQL Pool with Demo
Просмотров 4,4 тыс.5 месяцев назад
#AzureSynapse #SQLPools #InterviewPrep #DataAnalytics #CloudComputing #Azure #SynapseAnalytics #Serverless #DedicatedSQL #DataManagement #Database #DataWarehousing #BigData #TechExplainer #TechTutorial #TechEducation #TechInsights #DataScience In this video, we're diving deep into the world of Azure Synapse Analytics, focusing specifically on Serverless SQL Pools vs Dedicated SQL Pools. If you'...
Most popular Data Warehousing Interview Question- Azure Data Engineering Interview Question Tutorial
Просмотров 3,1 тыс.6 месяцев назад
#ApacheSpark #DataEngineering #AzureDataEngineer #SparkSQL #DataTransformation #DataFrame #InterviewQuestion #BigData #AzureDatabricks #PySpark #DataAnalysis #DataScience #SQLQuery #Optimization #Efficiency #Tutorial Welcome to our latest video where we delve into an essential interview question for Azure Data Engineers focusing on Type1 vs Type 2 in Data warehousing. As Data Engineers, it's re...
90% of Data Engineers doesn't know this - PySpark Azure Data Engineering Interview Question Tutorial
Просмотров 4,8 тыс.6 месяцев назад
#ApacheSpark #DataEngineering #AzureDataEngineer #SparkSQL #DataTransformation #DataFrame #InterviewQuestion #BigData #AzureDatabricks #PySpark #DataAnalysis #DataScience #SQLQuery #Optimization #Efficiency #Tutorial Welcome to our latest video where we delve into an essential interview question for Azure Data Engineers focusing on Apache Spark. Specifically, we tackle data transformation witho...
Build Your End-to-End Project in Just 3 Hours: Master Azure Data Engineering | Bing Data Analytics
Просмотров 19 тыс.6 месяцев назад
#AzureDataEngineering #MicrosoftFabric #DataFactory #SynapseAnalytics #DataIngestion #DataTransformation #DataScience #DataAnalytics In this video, I'll show you how to build a Bing News Data Analytics platform! We'll do a big project in Azure Data Engineering using Microsoft Fabric. We'll pull raw data from Bing API, transform the raw data to clean data with Synapse Data Engineering, analyze s...
Tips and Tricks- Azure Data Engineering Interview Questions | Managed Identity vs Service Principal
Просмотров 11 тыс.6 месяцев назад
Tips and Tricks- Azure Data Engineering Interview Questions | Managed Identity vs Service Principal
(Final) Part 9: Setting up Alerts using Data Activator and End to End Pipeline Testing | Azure Demo
Просмотров 2,7 тыс.7 месяцев назад
(Final) Part 9: Setting up Alerts using Data Activator and End to End Pipeline Testing | Azure Demo
Building Pipelines using Data Factory(Part-8)-End to End Azure Data Engineering Project using Fabric
Просмотров 3 тыс.7 месяцев назад
Building Pipelines using Data Factory(Part-8)-End to End Azure Data Engineering Project using Fabric
Building Reports using Power BI (Part-7) - End to End Azure Data Engineering Project using Fabric
Просмотров 3,2 тыс.7 месяцев назад
Building Reports using Power BI (Part-7) - End to End Azure Data Engineering Project using Fabric
Sentiment Analysis using Synapse ML(Part-6)- End to End Azure Data Engineering Project using Fabric
Просмотров 3,2 тыс.8 месяцев назад
Sentiment Analysis using Synapse ML(Part-6)- End to End Azure Data Engineering Project using Fabric
Incremental Load- Type 1(Part-5) - End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 4,7 тыс.8 месяцев назад
Incremental Load- Type 1(Part-5) - End to End Azure Data Engineering Project using Microsoft Fabric
Data Transformation (Part-4) - End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 6 тыс.8 месяцев назад
Data Transformation (Part-4) - End to End Azure Data Engineering Project using Microsoft Fabric
Data Ingestion (Part-3) - End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 7 тыс.8 месяцев назад
Data Ingestion (Part-3) - End to End Azure Data Engineering Project using Microsoft Fabric
(Part-2) Environment Setup- End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 9 тыс.8 месяцев назад
(Part-2) Environment Setup- End to End Azure Data Engineering Project using Microsoft Fabric
Bing News Data Analytics- End to End Azure Data Engineering Project using Microsoft Fabric
Просмотров 23 тыс.9 месяцев назад
Bing News Data Analytics- End to End Azure Data Engineering Project using Microsoft Fabric
A 6-Month Complete Guide to Become a Data Engineer in 2024 😎 - Azure Data Engineer Roadmap 2024
Просмотров 26 тыс.10 месяцев назад
A 6-Month Complete Guide to Become a Data Engineer in 2024 😎 - Azure Data Engineer Roadmap 2024
Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials
Просмотров 8 тыс.11 месяцев назад
Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials
Top 5 Notebook Features You Can't Miss in Azure Databricks 🚀| Databricks and Azure DevOps Tutorials
Просмотров 3,5 тыс.11 месяцев назад
Top 5 Notebook Features You Can't Miss in Azure Databricks 🚀| Databricks and Azure DevOps Tutorials
15 Minutes- Spark Clusters in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials
Просмотров 19 тыс.Год назад
15 Minutes- Spark Clusters in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials
Master Azure Databricks CI/CD in 2 Hours with Azure DevOps | Full End-to-End CI/CD Project in Azure
Просмотров 44 тыс.Год назад
Master Azure Databricks CI/CD in 2 Hours with Azure DevOps | Full End-to-End CI/CD Project in Azure

Комментарии

  • @parthiban.s4291
    @parthiban.s4291 13 часов назад

    Upload remaining part

  • @abdofathi9611
    @abdofathi9611 День назад

    The third part please

  • @XuanTran-t8t
    @XuanTran-t8t День назад

    Thank you for the video. May I ask if you runned this project with Pay-as-you-go or Free subscription?

  • @rajeshkilladi1826
    @rajeshkilladi1826 День назад

    "PERMISSION_DENIED: Invalid permissions on the specified KeyVault" is the error i came here for and thanks very much. It really helped

  • @billionairemindset7192
    @billionairemindset7192 2 дня назад

    Please drop the next video 😢😅, anticipation want to kill me.

  • @mujeebalam7885
    @mujeebalam7885 2 дня назад

    Brilliant Sir! it i something COMPLETE, really end to end. I am really enjoying watching this video.

  • @ZebanKarimSiddiqui
    @ZebanKarimSiddiqui 3 дня назад

    Can you please share the YML scrips. Or please share Github link for that.

  • @mainuser98
    @mainuser98 3 дня назад

    How to od this with workflows?

  • @SravanJangam
    @SravanJangam 3 дня назад

    waiting for next video, Can you please upload Microsoft fabric videos...

  • @SravanJangam
    @SravanJangam 3 дня назад

    Superb explanation. Can you please make a one how to handle real time streaming data using stream analytics with end to end project..

  • @RohitSharma-ny1oq
    @RohitSharma-ny1oq 3 дня назад

    great sir , Happy Diwali 🙂

  • @flyincarpet1338
    @flyincarpet1338 3 дня назад

    Mr.K can you please make a separate video for CICD integration in Fabric? specially for the free trial account

  • @KnightlyPursuits
    @KnightlyPursuits 4 дня назад

    It seems like nowadays it is not possible to create compute resources in databricks with free subscription. You need to have pay as you go subscription. Is it true or it's only happening for me.

  • @flyincarpet1338
    @flyincarpet1338 4 дня назад

    I like your statement "As you can see that " you have completed the final part :) Thanks a lot

  • @RajeshMaske-lo8zj
    @RajeshMaske-lo8zj 4 дня назад

    U r explaining like Robot...Everything is perfect.... Can u make this videos in Hindi

  • @praveen7307
    @praveen7307 4 дня назад

    bro when is part 3 rolling out

  • @flyincarpet1338
    @flyincarpet1338 4 дня назад

    Thank you very much, i also added the category filter so that i can only see the news like sports, politics etc and also check the sentiment on those verticles

  • @ashwingupta4765
    @ashwingupta4765 5 дней назад

    Awesome Explanation

  • @navaneeth44
    @navaneeth44 5 дней назад

    We can use df.groupby().agg() right?

  • @ATHARVA89
    @ATHARVA89 5 дней назад

    thanks!

  • @navaneeth44
    @navaneeth44 6 дней назад

    Great one, waiting for the next part in the series.

  • @sanjayjena984
    @sanjayjena984 6 дней назад

    Lots of learning from this session, its really helpful. Thank you so much Mr. K... your teaching style is unique and very clear in concept with simple business cases.👌

  • @flyincarpet1338
    @flyincarpet1338 7 дней назад

    crystal clear explanation :) Thanks

  • @Abhinavkumar-kt8gj
    @Abhinavkumar-kt8gj 7 дней назад

    Thank you so much for your videos. Excellent...!

  • @DrayCool-df8kj
    @DrayCool-df8kj 7 дней назад

    Thank you. Do you have a community ? I wanna join please.

  • @AlexFosterAI
    @AlexFosterAI 7 дней назад

    hey man, may be worth a shot checking out LakeSail's PySail built on rust. supposedly 4x faster with 90% less hardware a cost according to their latest benchmarks. and can migrate existing python code. might be cool to make a vid on! love ur content!

  • @AnghaKulkarni
    @AnghaKulkarni 7 дней назад

    You explained well about all concepts of Azure data factory, could please make more playlist for beginners from basic to advanced like SQL, pyspark, Azure storage gen2, required python concepts for data engineering, etc. So that we learn from basic to advance

  • @ravinithwin189
    @ravinithwin189 7 дней назад

    Big Shoutout to you brother, you just made the conceptual understanding easier ,keep going

  • @sandeepkumar-wy4hx
    @sandeepkumar-wy4hx 7 дней назад

    this video is savior for new aspirants

  • @pavankulkarni352
    @pavankulkarni352 7 дней назад

    This is the cleanest explanation I have ever come across on azure.

  • @sriharsha1761
    @sriharsha1761 8 дней назад

    you have already written the code and explaining it makes difficult to understand

  • @me.rajuofficial
    @me.rajuofficial 8 дней назад

    This tutorial is actually not for beginners Upload it properly how to install platforms to create those setup & all etc.

  • @julies5085
    @julies5085 8 дней назад

    How can we use Azure databricks in this project if we have the Trial version Azure Account(cluster running is prohibited in trial) ? Is there any other way? Can use Fabric for spark and azure functions?

  • @DhanasekharDanduprolu
    @DhanasekharDanduprolu 8 дней назад

    Pls do the video about SCD Type1 and SCD Type2 anna

  • @abdofathi9611
    @abdofathi9611 9 дней назад

    great video, thank u for your efforts, I can't wait for the next section,

  • @folorunsoajala1269
    @folorunsoajala1269 9 дней назад

    please always include code use in your video thanks

  • @digitalabi
    @digitalabi 9 дней назад

    I appreciate your explanation; it has clarified the topic for me. Thank you. 🙏🏼 However, I have one question: if the CSV files are split into two, how will one worker determine if there are any duplications with another worker work?

  • @VR-mq1ii
    @VR-mq1ii 10 дней назад

    Could you please tell us in how many parts you are going to finish this project ? Thanks

  • @ritwikpradhan8370
    @ritwikpradhan8370 10 дней назад

    Great help , thanks ✅

  • @priyeshudawat
    @priyeshudawat 10 дней назад

    Amazing content quality & explaination. Its the best data engineering project explained. Keep it up Sir.

  • @akashtribhuvan8124
    @akashtribhuvan8124 10 дней назад

    Thank you!

  • @DhanasekharDanduprolu
    @DhanasekharDanduprolu 11 дней назад

    Thank you anna you are my GURU❤️

  • @akashtribhuvan8124
    @akashtribhuvan8124 11 дней назад

    Thank you🙏

  • @akashtribhuvan8124
    @akashtribhuvan8124 11 дней назад

    Thank you!

  • @vinaydasari508
    @vinaydasari508 11 дней назад

    Mr.k ,Thanks for the project and expecting more end to end projects .

  • @bidata985
    @bidata985 11 дней назад

    Thank you for sharing valuable information and i really appreciate your efforts.

  • @snekhasp-sg9pw
    @snekhasp-sg9pw 12 дней назад

    Upload tamil

  • @sharaniyaswaminathan8760
    @sharaniyaswaminathan8760 12 дней назад

    This is great and very detailed!!!

  • @billionairemindset7192
    @billionairemindset7192 12 дней назад

    Thank you Mr KT

  • @joshh6683
    @joshh6683 12 дней назад

    Your videos are infinitely helpful! Thank you so much for all the hard work you put into them. Very useful!