Видео

Is this project available in udemy?

This project is indeed informative and well explained. Thank you.👍🏻

Hi Thanks for your video tutorial it's very insightful, I want to ask why we don't use fabric notebook instead of databriks notebook?

Imagine a Senior Data Engineer as the Senior Manager of a large department. They oversee the entire operation and have full control over all resources (the Azure Tenant). Within this department, the Senior Manager creates different teams (Resource Groups) to organize different services like Data Lake, Synapse Analytics, and Spark. To manage one of these teams, the Senior Manager appoints an Assistant Senior Manager as the team leader (Owner role) of a specific group. The Assistant Senior Manager has full control over the team and assigns roles to other members. For example, they may grant a Manager (Contributor role) the responsibility of managing the services, while a Team Leader or Team Member (Reader or Contributor roles) may be given specific access, like overseeing data ingestion or analysis. The Assistant Senior Manager ensures everyone has the appropriate access level to complete their tasks, while the Senior Manager maintains overall oversight of the department’s operations.

What is the difference between doing it in PySpark and SQL? I think in SQL it's much easier: %sql drop table if exists test; CREATE TABLE test ( id int, total int ) INSERT INTO test (id, total) VALUES (1,10), (1,20), (1,30), (1,40), (2,20), (2,40), (2,60), (2, 80); SELECT *, SUM(total) OVER (PARTITION BY id ORDER BY total) as emp_run FROM test;

hi, Can u make Video to cover Azure and Spark relates interview questions and answers wrt to real time scenarios focusing on optimization done in specific for the use case and not the general methadologies. These are the questions I was asked recently. 1) How do u recover a corrupt parquet data file 2) U have millions of records in bronze layer and after transformations u have 50 million records in gold layer. U find that there are corrupt files in only one partition at the gold layer. How will u recover the file of that particular partition without rerunning the entire pipeline because we have millions of rows in both bronze layers 3) What are the actual optimization done in project by you to achieve a) Execution time optimization b) Join level optimization Interviewer did not want generic answers which we know or would have read theortically. He wanted in specific How i implemented in the project Please do video with such tricky questions

Great video and project, but that Synapse part seems a bit irrelevant, or at least not worth of money; Isn't Synapse a bit overkill solution for loading data from lakehouse to database? why to use it instead of e.g. having a similar database in Azure and orchestrate with ADF+Databricks combo that was anyway used in the project?

One of the best training course. Thanks for covering all the steps. very useful for learning Azure Databricks

How can I follow your project when you have already set up eveyrthing?

Eagerly waiting for your next videos of this project

How can you execute for previous dates which is already gone.. we can just trigger for the old files, that we can do it with schedule trigger as well. its not like we can run the tumbling trigger for previous days..

This was much needed training, Thanks for sharing

nice project & clear explanation --> please continue to create this type of content

useful presentation

the end to end project is great explanation very useful content --- but while we are trying to do that from our end prerequisites is Advanture database with sample sales.lt tables on on-prem , those authentication process for new practitioners like me its difficult i request to do that video seperately to practise

Can we use UAMI - User assigned managed Identity to have connectivity in resources and what r its advantages and disadvantages and use case to use it

Hi sir , Why did you removed remaining parts of the playlist 😢 .... We didn't expect this from you .

dnt call it end-to-end project if you are skipping all of the initial steps you freakin' loser😡🤬

Nice video

Thank you for taking the time to create those wonderful videos. Can you tell us what would be the cost to complete all those training videos on a pay-as-you-go subscription? are there any resources that can be paused?

superb explanation loved it❤

Thanks for it. Please upload more videos

Thank you

AS AN FRESHSER WITH OR INTERN DE HOW TO WRITE SUCH CODES DO YOU HAVE ANY SOURCE WHERE WE CAN GET SUCH CODES ( I HEARD MANY DE DO COPY PASTE FROM AI OR GOOGLE JUST NEED TO KNOW LOGIC REPLACE FILES PATH)---BY SEEING SUCH HUGE CODES LOOK LIKE DE IS ALSO CODING----ITS DIFFICULT FOR NON IT GUYS IN ON PREM TO CLOUD U HAVE --CHANGES FOR DATES N COLUMN NAMES WHAT IN CASE THE REQUIREMENTS IS DIFFERENT LIKE ( REPLACE ALL TABLES NULL NOT NULL, REMOVE ALL TABLES BLANK SPACE, REMOVE ALL TABLESS DUPLICATES, ECT(ADD ON AS PER UR EXPERIENCE WHAT IS THE CODE FOR THAT???) IN BING API( U HAVE DONE TRANSFORMATION FOR OBJECT TO STRING N STING TO DISCTIONARY THEN MERGING FOR INCRIMENTAL LOAD --- WHAT IN CASE OF OTHER TRANSFORMATION REQUIED AS PER COMPANY -- HOW TO GET SUCH HUGE CODES? NOW THIS PROJECT( LOTS OF CODING---( KNOWING BASIC PYTHON N SQL IS OK BUT THIS LAST 2 PROJECT SERIES HAVE HUGE CODING -- HOW TO GET SUCH SCRIPTS PLEASE HELP)

Big fan sir

Very helpful!! 👍🏼

I have downloaded but I am unable to open the file

Superb explanation 👌 👏 👍 very helpful 👏 👍 👌

Thanks for your great explanation. Will keep watching your channel.

Download the code from here -> buymeacoffee.com/mrktalkstech/e/333857 Support me for my work :) Please Like, Share and Subscribe to my YT channel :)

Excellent 🙏🏻🙏🏻🙏🏻 ... Dedicated paid course is what we need from you SIR. You are great 🙏🏻🙏🏻🙏🏻

The Views in ASA are permanent? Or there are just virtual?

integration runtime Error Code: 1002 Error: A problem occurred while receiving the configuration file from the server. Suggestion: You may retry the express setup on Azure portal, or switch over to manual setup.

Please upload a dataset where we can download the dataset and can practise

Great Presentation! Just one suggestion for those working on real-time projects: If you're writing data into Parquet files and storing them in Azure Data Lake (which is essentially object storage at its core), here's something to consider. For example, if you have 1 million records in a Parquet file and a new record is added in the source, the system will update the entire 1 million records along with the new row. To prevent this issue, it's crucial to organize the data in Azure in a way that avoids large-scale updates. Instead of storing all sales data in a single folder, consider organizing it by year, then by month. If the dataset is extremely large, you can even break it down further by day. This way, when updates occur in the source, only a small part of the data is affected, rather than needing to update the entire dataset. /SalesData /Year=2023 /Month=01 /Day=01 sales_data_2023_01_01.parquet /Day=02 sales_data_2023_01_02.parquet /Month=02 /Day=01 sales_data_2023_02_01.parquet /Year=2024 /Month=01 /Day=01 sales_data_2024_01_01.parquet /Day=02 sales_data_2024_01_02.parquet

at 3:30 Where can I get my own password?

no dataset has been provided for practicing.

when is the next release

You are chandoo right??

waiting for videos seems like waiting for conformation of dhoni playing next ipl for chennai

Thank you very much for making this useful content! Shows your sincere effort.

good presentation

can any one help with sql store procedure query im getting below error: Started executing query at Line 1 Changed database context to 'gold_db'. (0 record affected) An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name. Total execution time: 00:00:01.645

can any one tell or wirte the sql store procedure query here

can you provide store procedure script all your videos of each projects series doesnt have scripts in discription im getting error for store procedure script

Respect++

@mr.ktalkstech when we can expect the part 2 video.

Brilliant man …. This is helping me a lot. Sidhi baat no bakwas hona chahiye is series a naam