@Dinesh Priyankara Quick question- I understand that U-SQL is used for data transformation tasks in ADL. If I want to transform data between Azure SQL database, what is our best options we have in Azure? I am from MS-BI background.
Hi Naveen, Sorry for not replying immediately. There are multiple options, since your both source and destinations are in Azure, you can use Azure Data Factory for transferring data between Azure databases. If you are planning to have all your Azure databases in an managed instance (still preview), then you can do the the transformation using TSQL as it supports cross-database queries.
may I know how you determine AU's required for a query so that they are best utilized? Also, how would I know if AU's are being underutilized in a query execution, is there any monitoring tool for U-SQL execution? Why Azure is making things so complex and have a bunch of options confusing the end users on what to pick depending on the use case scenario..in real world, I don't think you can design something for one scenario and then design some other thing for a different scenario..it does not work in reality and I don't see any videos talking about such things..all the videos are just focused on telling what the platform provides rather trying to solve actual real life situations..i would rather have the platform figure out how many AU's are required to execute rather than asking us to provide, that's a pain.
I am not sure whether I understand what you say, but if you say that videos published does not give any real world scenario, it is true because such thing cannot be easily made and I do not think that people who contribute have time to make a video on a project (that discusses a real-world scenario), it takes long time. And if you want to know how AUs have to be used, first execute the code using Visual Studio, once executed, you can see how they have been used, whether you have over used or need more. If you need to check AUs used for all jobs submitted, go to portal and see "Job Insights" in ADLA blade. It has all the details to determine how AUs have been used, whether you need more or not.
Hi Dinesh: No worries, your videos are absolutely helpful to understand the basics. I was looking for something more concrete real world example and I could not find even one that gives a problem statement, a solution and why such a solution approach was needed to solve such a problem statement. Probably, I myself do that one day. I also have one question for you in the mean time, can you help to answer the below question. As a data engineer, how would you design differently for real-time ETL compared to batch ETL? What additional steps you consider to design real-time ETL/ELT apart from capabilities (like scalability, resilience, etc )provided by the technology itself? How are the failures handled in real time ETL? Can you send your contact number to vins7983@gmail.com, i have few basic questions to ask.
it says that ruclips.net/user/pjHCGanZpDW is not a valid URL. Is the URL in your first slide is correct? Please let me know the URL for the ADLS/ADLA introduction video.
Superb Demonstration, for a beginner of U-SQL, this will be a good source.
I guess I'm pretty off topic but do anybody know a good website to watch new series online?
@Alexis Benson lately I have been using FlixZone. Just search on google for it :)
@Alexis Benson i would suggest Flixzone. You can find it by googling :)
Simply awesome Demos, Dinesh you are a star. Keep spreading knowledge.
Easy to understand, Excellent demo.
@Dinesh Priyankara Quick question- I understand that U-SQL is used for data transformation tasks in ADL. If I want to transform data between Azure SQL database, what is our best options we have in Azure? I am from MS-BI background.
Hi Naveen,
Sorry for not replying immediately.
There are multiple options, since your both source and destinations are in Azure, you can use Azure Data Factory for transferring data between Azure databases. If you are planning to have all your Azure databases in an managed instance (still preview), then you can do the the transformation using TSQL as it supports cross-database queries.
Hi Dinesh, the output files do not have any header; could you please tell how to add header to thw output files?
Any idea when ADLA will be supporting ADLS Gen 2?
@27:50, how have we READ 797 MB of file, *removed* headings, selected a sub-set of columns and then WRITTEN 953 MB !?!
good video.. thnks
How to load output data into a table on U- SQL Databasae instead of .CSV file?
is there anywhere i can get the presentation if you don't mind ..
may I know how you determine AU's required for a query so that they are best utilized? Also, how would I know if AU's are being underutilized in a query execution, is there any monitoring tool for U-SQL execution? Why Azure is making things so complex and have a bunch of options confusing the end users on what to pick depending on the use case scenario..in real world, I don't think you can design something for one scenario and then design some other thing for a different scenario..it does not work in reality and I don't see any videos talking about such things..all the videos are just focused on telling what the platform provides rather trying to solve actual real life situations..i would rather have the platform figure out how many AU's are required to execute rather than asking us to provide, that's a pain.
I am not sure whether I understand what you say, but if you say that videos published does not give any real world scenario, it is true because such thing cannot be easily made and I do not think that people who contribute have time to make a video on a project (that discusses a real-world scenario), it takes long time.
And if you want to know how AUs have to be used, first execute the code using Visual Studio, once executed, you can see how they have been used, whether you have over used or need more.
If you need to check AUs used for all jobs submitted, go to portal and see "Job Insights" in ADLA blade. It has all the details to determine how AUs have been used, whether you need more or not.
Hi Dinesh: No worries, your videos are absolutely helpful to understand the basics. I was looking for something more concrete real world example and I could not find even one that gives a problem statement, a solution and why such a solution approach was needed to solve such a problem statement. Probably, I myself do that one day. I also have one question for you in the mean time, can you help to answer the below question.
As a data engineer, how would you design differently for real-time ETL compared to batch ETL? What additional steps you consider to design real-time ETL/ELT apart from capabilities (like scalability, resilience, etc )provided by the technology itself? How are the failures handled in real time ETL? Can you send your contact number to vins7983@gmail.com, i have few basic questions to ask.
can I create tables in ADLA
Nice Video
it says that ruclips.net/user/pjHCGanZpDW is not a valid URL. Is the URL in your first slide is correct? Please let me know the URL for the ADLS/ADLA introduction video.
Hi Navin,
URL given is correct, it is ruclips.net/video/pjHcGanZpDw/видео.html, you have typed it wrong.
Thank you for correction.
how to update in datalake?
per-requisite video :- ruclips.net/video/pjHcGanZpDw/видео.html