Riz Ang
Riz Ang
  • Видео 47
  • Просмотров 301 176
What is Apache Avro file?
I will explain what Apache Avro is, the details under the hood and why you may want to consider using it.
0:00 Intro
0:37 Why consider other formats?
1:52 What is Avro?
2:41 Avro under the hood
4:39 Comparison between Avro and CSV
6:33 Should you use Avro?
8:03 Outro
Further readings:
- Avro official docs: avro.apache.org/docs/current/
- Data Serialization: docs.python-guide.org/scenarios/serialization/
- Avro for streaming: catherine-shen.medium.com/why-you-should-use-avro-and-schema-registry-for-your-streaming-application-2f24dcf017c8
Просмотров: 11 807

Видео

It’s All Analytics | Book Summary
Просмотров 4392 года назад
Book summary for “It’s All Analytics” by Scott Burk, Ph.D. and Gary D. Miner, Ph.D. Book Link: amzn.to/3IBPTLx 0:00 Introduction 0:24 First impression 1:20 Lesson 1: What is Analytics? 2:00 Lesson 2: The “4 happen” 3:18 Lesson 3: Detailed definitions 5:11 Lesson 4: How they fit together 6:13 Lesson 5: Data explosion 7:48 Lesson 6: Justifying analytics program 9:02 Lesson 7: People and Process, ...
Azure DevOps Pipeline Part 9 | How to setup DevOps self hosted agent
Просмотров 1,1 тыс.3 года назад
Part 9 video - Creating and setting up a new Azure DevOps self hosted agent Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment...
Azure DevOps Pipeline Part 8 | How to deploy Azure SQL Database with DevOps pipeline
Просмотров 4,6 тыс.3 года назад
Part 8 video - Deploying Azure SQL Database incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher e...
Azure DevOps Pipeline Part 7 | How to deploy Azure Databricks and Data Lake with DevOps pipeline
Просмотров 6 тыс.3 года назад
Part 7 video - Deploying Azure Databricks and Data Lake files incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or c...
Azure DevOps Pipeline Part 6 | How to deploy Azure Data Factory codes with DevOps pipeline
Просмотров 1,6 тыс.3 года назад
Part 6 video - Deploying Azure Data Factory codes incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into hi...
Azure DevOps Pipeline Part 5 | How to deploy Azure data platform with Terraform
Просмотров 1,4 тыс.3 года назад
Part 5 video - Deploy Azure data platform resources using Terraform and Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into...
Azure DevOps Pipeline Part 4 | Learn to deploy Azure resources with Terraform
Просмотров 1,2 тыс.3 года назад
Part 4 video - Learn the basics of Terraform and deploy Azure resources with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes...
Azure DevOps Pipeline Part 3 | How to deploy Azure resources with ARM template
Просмотров 2,7 тыс.3 года назад
Part 3 video - Deploying Azure resource group and blob storage with ARM template. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher envir...
Azure DevOps Pipeline Part 2 | How to create DevOps service connection
Просмотров 2,2 тыс.3 года назад
Part 2 video - Setting up DevOps pipeline service connection. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment. 0:00 Introdu...
Azure DevOps Pipeline Part 1 | How to deploy Azure Data Platform with DevOps pipeline
Просмотров 2,7 тыс.3 года назад
Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment. Part 1 video - Introduction and prerequisites 0:00 Introduction 0:52 Video...
What is Delta Lake? with Databricks
Просмотров 6 тыс.3 года назад
This video will cover what Delta Lake is about, their features and whether (or not) you may want to adopt. As always comes with a demo (feat Azure Databricks). Further reading: - delta.io - docs.delta.io/latest/index.html
Kappa Streaming Architecture in 6 minutes
Просмотров 1 тыс.3 года назад
Today's video will discuss a data processing architecture called Kappa Architecture. What is it, why it is used and how it may look.
How to extract Google Cloud Storage into Azure Data Lake (Data Factory)
Просмотров 4,1 тыс.3 года назад
Today's video will discuss how to copy a file from Google Cloud Storage to Azure Data Lake (gen2) with Azure Data Factory. Further reading: - docs.microsoft.com/en-us/azure/data-factory/connector-google-cloud-storage?tabs=data-factory
Lambda Architecture tutorial under 10 minutes
Просмотров 2,9 тыс.3 года назад
In today's video I will talk about lambda architecture, what it is and why it has been used, with some examples of latest technologies out there. Further reading: - en.wikipedia.org/wiki/Lambda_architecture
How to setup private endpoint with Azure Data Factory virtual network
Просмотров 10 тыс.3 года назад
How to setup private endpoint with Azure Data Factory virtual network
How to setup self hosted integration runtime in Azure Data Factory
Просмотров 1,8 тыс.3 года назад
How to setup self hosted integration runtime in Azure Data Factory
Azure Data Factory Debug VS. Trigger Explained
Просмотров 2 тыс.3 года назад
Azure Data Factory Debug VS. Trigger Explained
How to use Azure Data factory expressions (with examples!)
Просмотров 7 тыс.3 года назад
How to use Azure Data factory expressions (with examples!)
What is Apache Parquet file?
Просмотров 81 тыс.3 года назад
What is Apache Parquet file?
How to name pipelines / datasets / linked services in Azure Data Factory
Просмотров 2,1 тыс.3 года назад
How to name pipelines / datasets / linked services in Azure Data Factory
Pipeline Parameter vs. Variable in Azure Data Factory
Просмотров 7 тыс.3 года назад
Pipeline Parameter vs. Variable in Azure Data Factory
How to setup email alerts with Azure Log Analytics | Data Factory pipeline failures
Просмотров 11 тыс.3 года назад
How to setup email alerts with Azure Log Analytics | Data Factory pipeline failures
How to setup code repository in Azure Data Factory
Просмотров 6513 года назад
How to setup code repository in Azure Data Factory
Azure Data Lake Gen 2 VS. Azure Blob Storage Explained
Просмотров 30 тыс.3 года назад
Azure Data Lake Gen 2 VS. Azure Blob Storage Explained
How to extract SQL Database to Azure Data Lake gen 2 with data factory
Просмотров 6 тыс.3 года назад
How to extract SQL Database to Azure Data Lake gen 2 with data factory
Extract AWS S3 to Azure Data Lake gen 2 with Data Factory
Просмотров 5 тыс.3 года назад
Extract AWS S3 to Azure Data Lake gen 2 with Data Factory
How to pass Databricks exam | Associate Developer Spark 3.0
Просмотров 13 тыс.3 года назад
How to pass Databricks exam | Associate Developer Spark 3.0
What is the modern Data Analytics Platform (in 2021)
Просмотров 5953 года назад
What is the modern Data Analytics Platform (in 2021)
Setup pipeline alerts in Azure Data Factory
Просмотров 7 тыс.3 года назад
Setup pipeline alerts in Azure Data Factory

Комментарии

  • @Juan-Hdez
    @Juan-Hdez 7 дней назад

    Useful. Thank you.

  • @chethan4160
    @chethan4160 15 дней назад

    is it possible to movie data sql server to s3 using adf as i dont see any connector for s3 in sink

  • @hasnaa7316
    @hasnaa7316 17 дней назад

    very informative, thank you do much

  • @anamtarun6621
    @anamtarun6621 20 дней назад

    Hi riz, Directly i am connecting with report id but i can able to extract only 2000 records instead of extracting 1 lakh, do we have any solution on this. files

  • @kalhanganju2422
    @kalhanganju2422 Месяц назад

    Just one more thing, make sure you add a private endpoint to ADF resoirce before starting any of the steps.

  • @СергейСеливерстов-з2я

    500 лет рассказывал про 2+2, устал, даже досматривать не буду. Спасибо, как нибудь в другой раз.

  • @mestal
    @mestal 2 месяца назад

    Hi, there is a "Private endpoint" column in 06:00 and there is a link there. When we click this link, it is not being opened. There is a non existing subscription id in the link. Is this a bug? Do you know?

  • @CodingStyle-ii3iq
    @CodingStyle-ii3iq 2 месяца назад

    Thank you! Well done! very handy video.

  • @raghuvalab
    @raghuvalab 2 месяца назад

    Very helpful

  • @ramsvault
    @ramsvault 3 месяца назад

    thank you. wonderful explanation

  • @Adam-go5wv
    @Adam-go5wv 3 месяца назад

    I finally understand what is the parquet file format thanks to your video, great job!

  • @umasankar_4789
    @umasankar_4789 4 месяца назад

    Hi Riz. How can we extract the data from Salesforce marketing cloud by using azure fabrics

  • @raunakghosh7
    @raunakghosh7 4 месяца назад

    How to use custom domain?

  • @cusematt23
    @cusematt23 4 месяца назад

    thanks for the explanation. very nicely done.

  • @דורגולדשטיין-ד9ה
    @דורגולדשטיין-ד9ה 4 месяца назад

    Hi When clicking on Browse Sap Cubes - I cant see any cube opens ,empty list - eventhough "Test Connection" is succesful. why is that?

  • @kartikjaiswal8923
    @kartikjaiswal8923 5 месяцев назад

    crisp explanation! kudos!

  • @kuljotbakshi967
    @kuljotbakshi967 5 месяцев назад

    Great explanation!!!!

  • @owo4202
    @owo4202 5 месяцев назад

    Thanks for the clear explanation! It helps a lot!

  • @farzadshams3260
    @farzadshams3260 6 месяцев назад

    Thank you Riz. Very helpful video to get a high level understanding of the Parquet files!

    • @RizAngD
      @RizAngD 6 месяцев назад

      Glad to hear that!

  • @roadtrippingwithmihir
    @roadtrippingwithmihir 6 месяцев назад

    Excellent and crisp explanation

    • @RizAngD
      @RizAngD 6 месяцев назад

      Glad you liked it

  • @Anumin8
    @Anumin8 6 месяцев назад

    So you are not deploying the ADF code using Terraform, what is the reason? Is it because you cannot use the vsts configuration and deploy the code using Terraform at the same time?

  • @higiniofuentes2551
    @higiniofuentes2551 7 месяцев назад

    Thank you for this very useful video!

    • @RizAngD
      @RizAngD 6 месяцев назад

      Glad it was helpful!

  • @harryocallaghan6393
    @harryocallaghan6393 7 месяцев назад

    Really great explanation! thank you so much

    • @RizAngD
      @RizAngD 6 месяцев назад

      Glad you enjoyed it!

  • @ecmiguel
    @ecmiguel 8 месяцев назад

    Great!!!. Saludos desde Perú

    • @RizAngD
      @RizAngD 6 месяцев назад

      thanks!

  • @Afsarali-gm3sh
    @Afsarali-gm3sh 8 месяцев назад

    Hi Riz, how can i do it for all the pipelines, is that possible yes please tell me how can i achieve it in microsoft teams?

  • @multitaskprueba1
    @multitaskprueba1 8 месяцев назад

    You are a genius! Fantastic video! Thanks!

    • @RizAngD
      @RizAngD 6 месяцев назад

      Glad it helped!

  • @NitinKumar-td1wh
    @NitinKumar-td1wh 8 месяцев назад

    Hi Riz, Just a question abt the public IP address you allowed. As far as I understand Dynamics 365 uses multiple IP addresses. Is there a way to track the IP addresses we need to allow in Azure so we can add the entire range ? I have have searched but have not found a concrete Microsoft doc. Also this exposes the database on the public internet. Does D365 support any other tech like private link, endpoint etc ? P.S: I work in the infrastructure space and currently doing this for our D365 team. Appreciate your advice. Thanks in advance

  • @crixus3625
    @crixus3625 8 месяцев назад

    Thanks. How to test data migration to any free tool (and which one can you recommend)? Of course using Azure Data Factory as ETL

  • @repalasanthosh7452
    @repalasanthosh7452 9 месяцев назад

    How can we do this in way where we don’t want to create alert for each pipeline but want to monitor the future pipelines that will be deployed in ADF?

  • @MarkF-ix5mo
    @MarkF-ix5mo 9 месяцев назад

    Great video. Loved the fact that you used Physical Graffiti - one of my fave albums of all time.

    • @RizAngD
      @RizAngD 6 месяцев назад

      thanks!!

  • @Vmr48765
    @Vmr48765 9 месяцев назад

    what if i want to download the entire folder i.e. all files into ADSLv2?

  • @devarapallivamsi7064
    @devarapallivamsi7064 10 месяцев назад

    Good and to the point.

    • @RizAngD
      @RizAngD 6 месяцев назад

      thanks!

  • @nagamanickam6604
    @nagamanickam6604 10 месяцев назад

    Thank you very much

    • @RizAngD
      @RizAngD 6 месяцев назад

      You are welcome

  • @meghakumari1506
    @meghakumari1506 10 месяцев назад

    How can we bring D365fo tables like this?

  • @nehashahpatel1741
    @nehashahpatel1741 10 месяцев назад

    Thanks

    • @RizAngD
      @RizAngD 6 месяцев назад

      Welcome

  • @fuzzy93
    @fuzzy93 11 месяцев назад

    How can i use this same method to connect to a private endpoint resource provided external to my organization/subscription?

  • @sukumarmusalaboina3375
    @sukumarmusalaboina3375 11 месяцев назад

    Thanks for the video .. How to automate this process or doing it in ADF completely.

    • @UnbelievableOdyssey
      @UnbelievableOdyssey 10 месяцев назад

      You'd need to use Data Flows. They support CDM natively.

  • @paul1113-zw5pn
    @paul1113-zw5pn Год назад

    Very well explained Encoding and Compression...So I have a Q: Delta versus Dictionary Encoding, How would one decide which given Dictionary seems so much more efficient? But then I suppose it depends on repitition.

  • @Tnradar
    @Tnradar Год назад

    Why can't I have Databricks in place of Tableau as well?

  • @AshokG12
    @AshokG12 Год назад

    we are trying to create servicenow ticket from log anlytics in case there is any failure. we want to send selected fields like error message. can we do that. either using logic app or through ITSM connector. i did not find any way to send query columns to any of the action groups.

  • @ValenteArellanoMartinez
    @ValenteArellanoMartinez Год назад

    very useful, can you share cost for that operation in azure? thanks

  • @HamzaMediani
    @HamzaMediani Год назад

    Hi Riz, do you know how i can overwrite my linkedsrvices/dataets cedentials. i'm deploying from dev to production and the credentials are different

  • @w9621997
    @w9621997 Год назад

    Hi! Very good video. One question; is a new Blob container always created automatically?

  • @sathyanarayanareddy5192
    @sathyanarayanareddy5192 Год назад

    Very good it helped me

  • @sheheryar89
    @sheheryar89 Год назад

    Thanks

  • @nagsworld
    @nagsworld Год назад

    How can we automate this process?

  • @taglud
    @taglud Год назад

    good and simple explanation, thank you :)

  • @ravitalaviya1576
    @ravitalaviya1576 Год назад

    I am currently capturing live data in csv format. But for storage benefit, i want to live data is saved in direct parquet format. that is possible or not?

  • @nikjojo
    @nikjojo Год назад

    great simple video thanks

  • @abhishekchaudhary3597
    @abhishekchaudhary3597 Год назад

    I have successfully copied data from Salesforce to my Azure SQL Database using Azure Data Factory pipelines, and I want to ensure that my pipeline automatically retrieves updated or new entries from Salesforce into my database. I am considering using Change Data Capture event from Salesforce and subscribing to the event using Azure Function or Event Hub. Can someone advise me on the best way to achieve this? Additionally, I want to ensure that any deleted rows in Salesforce are updated in my Azure SQL database.