- Видео 47
- Просмотров 301 176
Riz Ang
Индонезия
Добавлен 27 май 2020
Hello! I'm Riz and I am a hybrid data product owner / data engineer / data analyst.
My channel is all about building and delivering high value data products for businesses.
Subscribe to my channel if you find my videos helpful!
My channel is all about building and delivering high value data products for businesses.
Subscribe to my channel if you find my videos helpful!
What is Apache Avro file?
I will explain what Apache Avro is, the details under the hood and why you may want to consider using it.
0:00 Intro
0:37 Why consider other formats?
1:52 What is Avro?
2:41 Avro under the hood
4:39 Comparison between Avro and CSV
6:33 Should you use Avro?
8:03 Outro
Further readings:
- Avro official docs: avro.apache.org/docs/current/
- Data Serialization: docs.python-guide.org/scenarios/serialization/
- Avro for streaming: catherine-shen.medium.com/why-you-should-use-avro-and-schema-registry-for-your-streaming-application-2f24dcf017c8
0:00 Intro
0:37 Why consider other formats?
1:52 What is Avro?
2:41 Avro under the hood
4:39 Comparison between Avro and CSV
6:33 Should you use Avro?
8:03 Outro
Further readings:
- Avro official docs: avro.apache.org/docs/current/
- Data Serialization: docs.python-guide.org/scenarios/serialization/
- Avro for streaming: catherine-shen.medium.com/why-you-should-use-avro-and-schema-registry-for-your-streaming-application-2f24dcf017c8
Просмотров: 11 807
Видео
It’s All Analytics | Book Summary
Просмотров 4392 года назад
Book summary for “It’s All Analytics” by Scott Burk, Ph.D. and Gary D. Miner, Ph.D. Book Link: amzn.to/3IBPTLx 0:00 Introduction 0:24 First impression 1:20 Lesson 1: What is Analytics? 2:00 Lesson 2: The “4 happen” 3:18 Lesson 3: Detailed definitions 5:11 Lesson 4: How they fit together 6:13 Lesson 5: Data explosion 7:48 Lesson 6: Justifying analytics program 9:02 Lesson 7: People and Process, ...
Azure DevOps Pipeline Part 9 | How to setup DevOps self hosted agent
Просмотров 1,1 тыс.3 года назад
Part 9 video - Creating and setting up a new Azure DevOps self hosted agent Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment...
Azure DevOps Pipeline Part 8 | How to deploy Azure SQL Database with DevOps pipeline
Просмотров 4,6 тыс.3 года назад
Part 8 video - Deploying Azure SQL Database incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher e...
Azure DevOps Pipeline Part 7 | How to deploy Azure Databricks and Data Lake with DevOps pipeline
Просмотров 6 тыс.3 года назад
Part 7 video - Deploying Azure Databricks and Data Lake files incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or c...
Azure DevOps Pipeline Part 6 | How to deploy Azure Data Factory codes with DevOps pipeline
Просмотров 1,6 тыс.3 года назад
Part 6 video - Deploying Azure Data Factory codes incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into hi...
Azure DevOps Pipeline Part 5 | How to deploy Azure data platform with Terraform
Просмотров 1,4 тыс.3 года назад
Part 5 video - Deploy Azure data platform resources using Terraform and Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into...
Azure DevOps Pipeline Part 4 | Learn to deploy Azure resources with Terraform
Просмотров 1,2 тыс.3 года назад
Part 4 video - Learn the basics of Terraform and deploy Azure resources with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes...
Azure DevOps Pipeline Part 3 | How to deploy Azure resources with ARM template
Просмотров 2,7 тыс.3 года назад
Part 3 video - Deploying Azure resource group and blob storage with ARM template. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher envir...
Azure DevOps Pipeline Part 2 | How to create DevOps service connection
Просмотров 2,2 тыс.3 года назад
Part 2 video - Setting up DevOps pipeline service connection. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment. 0:00 Introdu...
Azure DevOps Pipeline Part 1 | How to deploy Azure Data Platform with DevOps pipeline
Просмотров 2,7 тыс.3 года назад
Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment. Part 1 video - Introduction and prerequisites 0:00 Introduction 0:52 Video...
What is Delta Lake? with Databricks
Просмотров 6 тыс.3 года назад
This video will cover what Delta Lake is about, their features and whether (or not) you may want to adopt. As always comes with a demo (feat Azure Databricks). Further reading: - delta.io - docs.delta.io/latest/index.html
Kappa Streaming Architecture in 6 minutes
Просмотров 1 тыс.3 года назад
Today's video will discuss a data processing architecture called Kappa Architecture. What is it, why it is used and how it may look.
How to extract Google Cloud Storage into Azure Data Lake (Data Factory)
Просмотров 4,1 тыс.3 года назад
Today's video will discuss how to copy a file from Google Cloud Storage to Azure Data Lake (gen2) with Azure Data Factory. Further reading: - docs.microsoft.com/en-us/azure/data-factory/connector-google-cloud-storage?tabs=data-factory
Lambda Architecture tutorial under 10 minutes
Просмотров 2,9 тыс.3 года назад
In today's video I will talk about lambda architecture, what it is and why it has been used, with some examples of latest technologies out there. Further reading: - en.wikipedia.org/wiki/Lambda_architecture
How to setup private endpoint with Azure Data Factory virtual network
Просмотров 10 тыс.3 года назад
How to setup private endpoint with Azure Data Factory virtual network
How to setup self hosted integration runtime in Azure Data Factory
Просмотров 1,8 тыс.3 года назад
How to setup self hosted integration runtime in Azure Data Factory
Azure Data Factory Debug VS. Trigger Explained
Просмотров 2 тыс.3 года назад
Azure Data Factory Debug VS. Trigger Explained
How to use Azure Data factory expressions (with examples!)
Просмотров 7 тыс.3 года назад
How to use Azure Data factory expressions (with examples!)
How to name pipelines / datasets / linked services in Azure Data Factory
Просмотров 2,1 тыс.3 года назад
How to name pipelines / datasets / linked services in Azure Data Factory
Pipeline Parameter vs. Variable in Azure Data Factory
Просмотров 7 тыс.3 года назад
Pipeline Parameter vs. Variable in Azure Data Factory
How to setup email alerts with Azure Log Analytics | Data Factory pipeline failures
Просмотров 11 тыс.3 года назад
How to setup email alerts with Azure Log Analytics | Data Factory pipeline failures
How to setup code repository in Azure Data Factory
Просмотров 6513 года назад
How to setup code repository in Azure Data Factory
Azure Data Lake Gen 2 VS. Azure Blob Storage Explained
Просмотров 30 тыс.3 года назад
Azure Data Lake Gen 2 VS. Azure Blob Storage Explained
How to extract SQL Database to Azure Data Lake gen 2 with data factory
Просмотров 6 тыс.3 года назад
How to extract SQL Database to Azure Data Lake gen 2 with data factory
Extract AWS S3 to Azure Data Lake gen 2 with Data Factory
Просмотров 5 тыс.3 года назад
Extract AWS S3 to Azure Data Lake gen 2 with Data Factory
How to pass Databricks exam | Associate Developer Spark 3.0
Просмотров 13 тыс.3 года назад
How to pass Databricks exam | Associate Developer Spark 3.0
What is the modern Data Analytics Platform (in 2021)
Просмотров 5953 года назад
What is the modern Data Analytics Platform (in 2021)
Setup pipeline alerts in Azure Data Factory
Просмотров 7 тыс.3 года назад
Setup pipeline alerts in Azure Data Factory
Useful. Thank you.
is it possible to movie data sql server to s3 using adf as i dont see any connector for s3 in sink
very informative, thank you do much
Hi riz, Directly i am connecting with report id but i can able to extract only 2000 records instead of extracting 1 lakh, do we have any solution on this. files
Just one more thing, make sure you add a private endpoint to ADF resoirce before starting any of the steps.
500 лет рассказывал про 2+2, устал, даже досматривать не буду. Спасибо, как нибудь в другой раз.
Hi, there is a "Private endpoint" column in 06:00 and there is a link there. When we click this link, it is not being opened. There is a non existing subscription id in the link. Is this a bug? Do you know?
Thank you! Well done! very handy video.
Very helpful
thank you. wonderful explanation
I finally understand what is the parquet file format thanks to your video, great job!
Hi Riz. How can we extract the data from Salesforce marketing cloud by using azure fabrics
How to use custom domain?
thanks for the explanation. very nicely done.
Hi When clicking on Browse Sap Cubes - I cant see any cube opens ,empty list - eventhough "Test Connection" is succesful. why is that?
crisp explanation! kudos!
Great explanation!!!!
Thanks for the clear explanation! It helps a lot!
Thank you Riz. Very helpful video to get a high level understanding of the Parquet files!
Glad to hear that!
Excellent and crisp explanation
Glad you liked it
So you are not deploying the ADF code using Terraform, what is the reason? Is it because you cannot use the vsts configuration and deploy the code using Terraform at the same time?
Thank you for this very useful video!
Glad it was helpful!
Really great explanation! thank you so much
Glad you enjoyed it!
Great!!!. Saludos desde Perú
thanks!
Hi Riz, how can i do it for all the pipelines, is that possible yes please tell me how can i achieve it in microsoft teams?
You are a genius! Fantastic video! Thanks!
Glad it helped!
Hi Riz, Just a question abt the public IP address you allowed. As far as I understand Dynamics 365 uses multiple IP addresses. Is there a way to track the IP addresses we need to allow in Azure so we can add the entire range ? I have have searched but have not found a concrete Microsoft doc. Also this exposes the database on the public internet. Does D365 support any other tech like private link, endpoint etc ? P.S: I work in the infrastructure space and currently doing this for our D365 team. Appreciate your advice. Thanks in advance
Thanks. How to test data migration to any free tool (and which one can you recommend)? Of course using Azure Data Factory as ETL
How can we do this in way where we don’t want to create alert for each pipeline but want to monitor the future pipelines that will be deployed in ADF?
Great video. Loved the fact that you used Physical Graffiti - one of my fave albums of all time.
thanks!!
what if i want to download the entire folder i.e. all files into ADSLv2?
Good and to the point.
thanks!
Thank you very much
You are welcome
How can we bring D365fo tables like this?
Thanks
Welcome
How can i use this same method to connect to a private endpoint resource provided external to my organization/subscription?
Thanks for the video .. How to automate this process or doing it in ADF completely.
You'd need to use Data Flows. They support CDM natively.
Very well explained Encoding and Compression...So I have a Q: Delta versus Dictionary Encoding, How would one decide which given Dictionary seems so much more efficient? But then I suppose it depends on repitition.
Why can't I have Databricks in place of Tableau as well?
we are trying to create servicenow ticket from log anlytics in case there is any failure. we want to send selected fields like error message. can we do that. either using logic app or through ITSM connector. i did not find any way to send query columns to any of the action groups.
very useful, can you share cost for that operation in azure? thanks
Hi Riz, do you know how i can overwrite my linkedsrvices/dataets cedentials. i'm deploying from dev to production and the credentials are different
Hi! Very good video. One question; is a new Blob container always created automatically?
Very good it helped me
Thanks
How can we automate this process?
good and simple explanation, thank you :)
I am currently capturing live data in csv format. But for storage benefit, i want to live data is saved in direct parquet format. that is possible or not?
great simple video thanks
I have successfully copied data from Salesforce to my Azure SQL Database using Azure Data Factory pipelines, and I want to ensure that my pipeline automatically retrieves updated or new entries from Salesforce into my database. I am considering using Change Data Capture event from Salesforce and subscribing to the event using Azure Function or Event Hub. Can someone advise me on the best way to achieve this? Additionally, I want to ensure that any deleted rows in Salesforce are updated in my Azure SQL database.