Data Tech
Data Tech
  • Видео 80
  • Просмотров 757 586
dbt + Airflow = ❤ ; An open source project that integrates dbt and Airflow
In this video, we will learn how to run dbt-core jobs in airflow using an open source project.
▬▬▬▬▬▬ Links 🔗 ▬▬▬▬▬▬
► Project : github.com/AnandDedha/dbt-airflow-demo
► How to run the project locally : github.com/AnandDedha/dbt-airflow-demo/blob/main/docs/how_to_run_this_project.md
► Official Documentation - gmyrianthous.github.io/dbt-airflow/
► How to contribute to this open source project - github.com/gmyrianthous/dbt-airflow/blob/main/docs/contributing.md
▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
00:00 Getting Started
02:51 Project Architecture
07:53 Project folder Structure
11:15 Docker file and docker compose
20:43 Running Docker Container to set up the project
34:11 Airflow Dags
51:54 How to integr...
Просмотров: 10 453

Видео

dbt(Data Build Tool) macros crash course: Zero to Hero | Jinja role in dbt | Third-party macros
Просмотров 2,2 тыс.10 месяцев назад
In this video tutorial, we will learn about dbt (data build tool) macros, we'll see how Jinja helps make SQL more dynamic and also step by step how to install third party macros in dbt, which add more features to your dbt projects. ▬▬▬▬▬▬ Links 🔗 ▬▬▬▬▬▬ ► docs.getdbt.com/docs/build/jinja-macros ► github.com/AnandDedha/dbt-bq-demo/tree/main ►github.com/AnandDedha/dbt-bq-demo/blob/main/docs/jinja...
dbt(Data Build Tool) crash course for beginners: Zero to Hero
Просмотров 48 тыс.Год назад
In this video tutorial, we will learn about dbt (data build tool), the core concepts of dbt, exploring its project structure and key components. It will guide us through setting up dbt Cloud using BigQuery and GitHub. Additionally, the tutorial covers various topics of dbt such as models, building & running them, understanding Macros, Generic and Singular Tests, and Snapshots within dbt. ▬▬▬▬▬▬...
Apache Kafka tutorial for beginners using food delivery apps such as UberEats or Swiggy
Просмотров 1,1 тыс.Год назад
In this video, we'll dive into the fundamental concepts of Kafka by relating it to familiar food delivery apps like UberEats or Swiggy. We'll explore what Kafka actually is, why it has gained so much popularity, and get a clear understanding of how it functions behind the scenes. ppt Github - github.com/AnandDedha/apache/blob/main/Apache Kafka.pdf #KafkaBeginnersGuide #KafkaTutorial #LearnKafka...
How to build and automate a ETL pipeline with AWS airflow | AWS End-To-End Data Engineering Project
Просмотров 16 тыс.Год назад
In this data engineering project,we're creating a data pipeline on Amazon Web Services (AWS) using airflow, python, spark, Glue, Redshift and other AWS services. We will learn how to build and automate an ETL process that that can extract the weather data from open weather map API, transform the data using spark and load the data into Redshift using Apache Airflow. Here, the necessary infrastru...
AWS Athena Tutorial with SQL & Pyspark l Athena Hands On LAB | Athena + Glue + S3 Data Lake
Просмотров 3,7 тыс.Год назад
In this video, we learn about the Athena. 1. What is Athena? 2. How AWS Athena works? 3. When it should be used ? 4. How can interactive SQL Queries be executed within the Athena editor? 5. How can a notebook be configured and pyspark interactively utilised within Athena? #aws #awsdataengineer #awsdataanalytics #awsbigdata #AWSDataEngineering #awstraining #awscloudpractitioner #awsclouddataengi...
End-To-End Data Engineering Project in AWS | Build complete Data Pipeline in AWS within 25 mins
Просмотров 6 тыс.Год назад
Architecture template link - github.com/AnandDedha/AWS/blob/main/aws-etl/s3-glue-redshift-iam.yaml Pyspark script link - github.com/AnandDedha/AWS/blob/main/aws-etl/s3-glue-redshift-iam.yaml Data sample link - github.com/AnandDedha/AWS/blob/main/aws-etl/sales_records.csv 00:42 Data Engineering architecture in AWS 04:55 Data-pipeline using Pyspark in AWS 11:21 ETL code explanation 22:51 Summary ...
AWS data engineering architecture setup with just 2 clicks| AWS ETL Infrastructure complete setup
Просмотров 455Год назад
Data Engineering Architecture Template- github.com/AnandDedha/AWS/blob/main/aws-etl/s3-glue-redshift.yaml This template creates end-end AWS ETL infrastructure which includes the VPC, Internet Gateway, VPC Gateway Attachment, and various subnets for me. It even sets up NAT gateways for public subnets, route tables, VPC endpoints for S3 access, S3 buckets, security groups for Redshift, IAM role f...
Learn how to load data into DynamoDB using python from AWS S3
Просмотров 2,3 тыс.Год назад
In this video, we will explore a step-by-step guide on utilizing a Python Lambda function to seamlessly import JSON data from an S3 bucket into DynamoDB. By following along with the provided example, you will gain a comprehensive understanding of how to achieve this data transfer seamlessly. The video will cover the following key steps: Lambda Function Creation: We will guide you through the pr...
Learn AWS services of data engineering in 10 mins
Просмотров 877Год назад
As a data engineer working with AWS, knowing the following top AWS services is crucial to design, build, and maintain data pipelines, databases, and analytics solutions: Amazon S3 (Simple Storage Service): ruclips.net/video/jWtf3LQTdr8/видео.html AWS Glue: ruclips.net/video/coJhlQlgHVk/видео.html ruclips.net/video/BNsE3qKtA2w/видео.html AWS Lambda: ruclips.net/video/mGn2XTuFvas/видео.html rucli...
Simplified step-by-step process for beginners to understand how AWS Lambda triggers work with events
Просмотров 600Год назад
In this video, you'll find comprehensive tutorial and step-by-step guide that walk you through various aspects of AWS Lambda triggers with events #awslambda #lambda #aws #awsdataengineer #awsbigdata #serverless #python Amazon Lambda AWS Big Data AWS Data Engineer AWS Data Analytics AWS Python AWS Server-less Lambda Function GitHub Link - github.com/AnandDedha/AWS/blob/main/Lamda/s3event.py S3 E...
Introduction to AWS Lambda with hands on demo | AWS lambda tutorial for beginners within 10 mins
Просмотров 2,1 тыс.Год назад
In this video, we discuss about the basic structure of AWS lambda, AWS Lambda is a server-less compute service provided by Amazon Web Services (AWS). It allows you to run your code without provisioning or managing servers. You can write your Lambda functions in various programming languages, including Python. A Lambda function is triggered by an event and executes a specific piece of code in re...
Learn how to create DynamoDB table using AWS Console | Build first DynamoDB table
Просмотров 2,8 тыс.Год назад
This video will teach us the process of creating a DynamoDB table on AWS. It will cover the steps to create a table with a primary key and indexes. The lesson also covers the concepts of querying and scanning. #dynamodb #AWSCertifiedDataAnalyticsSpecialty #AWS #AWSDAS-C01 #AWSDataEngineer #AWSDataAnalytics #DataAnalyticsSpecialty #AWSBigData Amazon DynamoDB AWS Certified Data Analytics Specialt...
DynamoDB Local and Global Secondary Indexes: Improve Query Performance and Flexibility
Просмотров 633Год назад
This video covers the topic of DynamoDB indexes, focusing on both Global Secondary Index (GSI) and Local Secondary Index (LSI) #dynamodb #AWSCertifiedDataAnalyticsSpecialty #AWS #AWSDAS-C01 #AWSDataEngineer #AWSDataAnalytics #DataAnalyticsSpecialty #AWSBigData Amazon DynamoDB AWS Certified Data Analytics Specialty AWS Certified Data Analytics - Specialty (DAS-C01) Exam Guide AWS Big Data AWS Da...
Introduction to AWS DynamoDB | Beginners guide for AWS DynamoDB
Просмотров 959Год назад
In this video, we learn about Dynamodb introduction and DynamoDB Core Concepts: 1. Tables: DynamoDB organizes data into tables, which are similar to tables in a relational database. Each table consists of multiple items, and each item is uniquely identified by a primary key. 2.Items: An item is a collection of attributes that represents a single data record in DynamoDB. Each item is identified ...
Learn how to perform ETL & Cataloging on the data using AWS Glue | Build Data Pipeline using Glue
Просмотров 845Год назад
Learn how to perform ETL & Cataloging on the data using AWS Glue | Build Data Pipeline using Glue
AWS Glue tutorial for beginners| AWS Concepts that all you need to know
Просмотров 356Год назад
AWS Glue tutorial for beginners| AWS Concepts that all you need to know
Amazon/AWS VPC (Virtual Private Cloud) Basics | AWS VPC Tutorial for Beginners/Non -Network Folks
Просмотров 544Год назад
Amazon/AWS VPC (Virtual Private Cloud) Basics | AWS VPC Tutorial for Beginners/Non -Network Folks
Amazon Redshift Operations - Utilizing Vacuum & Deep Copy
Просмотров 1,9 тыс.Год назад
Amazon Redshift Operations - Utilizing Vacuum & Deep Copy
Data Engineering resume tips for landing more interviews
Просмотров 1 тыс.Год назад
Data Engineering resume tips for landing more interviews
Redshift Spectrum Explained: Querying S3 without loading into Redshift
Просмотров 7 тыс.Год назад
Redshift Spectrum Explained: Querying S3 without loading into Redshift
Amazon Redshift - A Beginner's Guide to Cloud Data Warehousing of Redshift Clusters & Server-less
Просмотров 8 тыс.Год назад
Amazon Redshift - A Beginner's Guide to Cloud Data Warehousing of Redshift Clusters & Server-less
Introduction to Amazon Relational Database Service (RDS) for beginners
Просмотров 3,5 тыс.Год назад
Introduction to Amazon Relational Database Service (RDS) for beginners
AWS S3 Tutorial (Part 6) - AWS Hands on Lab Amazon S3 - Object Lock
Просмотров 1,1 тыс.Год назад
AWS S3 Tutorial (Part 6) - AWS Hands on Lab Amazon S3 - Object Lock
AWS S3 Tutorial (Part 5) - S3 Life Cycle Management
Просмотров 1,2 тыс.Год назад
AWS S3 Tutorial (Part 5) - S3 Life Cycle Management
AWS S3 Tutorial (Part 4) - Amazon S3 Versioning & Replication
Просмотров 1,5 тыс.Год назад
AWS S3 Tutorial (Part 4) - Amazon S3 Versioning & Replication
AWS S3 Tutorial (Part 3) - How to set up Access Control on S3 ? IAM Policies & Bucket Policies.
Просмотров 2,6 тыс.Год назад
AWS S3 Tutorial (Part 3) - How to set up Access Control on S3 ? IAM Policies & Bucket Policies.
AWS S3 Tutorial(Part 2) - How to configure AWS account with CLI & how to get data into S3 using CLI
Просмотров 3,3 тыс.Год назад
AWS S3 Tutorial(Part 2) - How to configure AWS account with CLI & how to get data into S3 using CLI
AWS S3 Tutorial (Part1) - Introduction to Amazon S3 (Simple Storage Service)
Просмотров 8 тыс.Год назад
AWS S3 Tutorial (Part1) - Introduction to Amazon S3 (Simple Storage Service)
AWS Certified Data Analytics - Specialty (DAS-C01) Exam Overview
Просмотров 19 тыс.Год назад
AWS Certified Data Analytics - Specialty (DAS-C01) Exam Overview

Комментарии

  • @aishmash5325
    @aishmash5325 17 часов назад

    These videos are just awesome!😇 Thank you so much.

  • @Vikasptl07
    @Vikasptl07 3 дня назад

    DBT for me is just a tool that fills the gap where traditional SQL world was lacking in terms of some unit testing ( but much of it depends on the test Data you create which should be as close as your source which is quite impossible) and documentation capability. If you are starting with Data warehousing then this can be good choice. This is not a tool for DE. if you are good at SQL and don't want another wrappper to manage your views then you can skip this.

  • @rohitpant6473
    @rohitpant6473 8 дней назад

    This video was of no use.

  • @ManikantaKorada-b9u
    @ManikantaKorada-b9u 11 дней назад

    Very helpful. Thanks

  • @shubhammahajan9117
    @shubhammahajan9117 13 дней назад

    Hey man, extremely useful content. Just a quick question, how is cosmos different from this? Cosmos with airflow also helps to achieve the same,right?

  • @arunan8922
    @arunan8922 26 дней назад

    Excellent Video

  • @aviralsatija6395
    @aviralsatija6395 Месяц назад

    Very helpful. Thanks for this anand

  • @NithyashreeBalaji-y8t
    @NithyashreeBalaji-y8t Месяц назад

    Hello, Thank you for the AWS Glue hands on. do we have a dataset link for this?

  • @meganlafleur1781
    @meganlafleur1781 Месяц назад

    The only video i've found explaining dbt like a human. Thank you so much.

  • @s_m_exist
    @s_m_exist 2 месяца назад

    Thank you for the video!, 14:13 How does enterprise does this connection without service account?

  • @govind1973
    @govind1973 2 месяца назад

    Great.

  • @edydon
    @edydon 2 месяца назад

    Very clear and helpful. Keep up the great work!

  • @zoro-nw6jx
    @zoro-nw6jx 2 месяца назад

    thanks best video learned a lot

  • @sathyas179
    @sathyas179 2 месяца назад

    According to the explanation that you gave, tumbling window is non-overlapping window function but you have categorized that into overlapping function.

  • @ravirajasekharuni
    @ravirajasekharuni 3 месяца назад

    Very good information and practical steps

  • @CarbonsHDTuts
    @CarbonsHDTuts 3 месяца назад

    What file format is this ? Parquet?

  • @junpei0berkeley
    @junpei0berkeley 3 месяца назад

    well done!

  • @abdulazizmamun
    @abdulazizmamun 3 месяца назад

    Great Video Anand. Time to make hand dirty

  • @NatTata-x1q
    @NatTata-x1q 4 месяца назад

    Great topic and great explanation! Hope to see the examples you mentioned in this video! Thanks!

  • @NarendraKumar-kb1vs
    @NarendraKumar-kb1vs 4 месяца назад

    Thank you so much for your tutorials, this really helps me to start dbt. Please share your email id to contact for training.

  • @arjunkumar-ff5bf
    @arjunkumar-ff5bf 4 месяца назад

    This was the worst tutorial ever. Please teach properly or stop teaching at all.

  • @Ahmed-lt6rr
    @Ahmed-lt6rr 4 месяца назад

    Do you have any platform or discord ?

  • @VishalDubey-k3e
    @VishalDubey-k3e 4 месяца назад

    supper video sir💓❤❤ I like your way of explanation

  • @tharunk8019
    @tharunk8019 5 месяцев назад

    Can you pls increase the video quality or resolution?

  • @iExplorer64
    @iExplorer64 5 месяцев назад

    wait, so we need linux to be a data engineer? we cant use windows?

  • @dominicaleung7329
    @dominicaleung7329 5 месяцев назад

    thank you very much for your tutorial. very nice. Very good pace.

  • @tastykhaana8999
    @tastykhaana8999 5 месяцев назад

    Not a very good explanation.

  • @MadhavCloud
    @MadhavCloud 5 месяцев назад

    Nice video and was very helpful understand both dbt as well as bigquery along with Git integration but looks like missed something, I did not understand how the folder dbt_packages appeared in dbt and how raw folder was created and files uploaded. any guidance is much appreciated.

  • @jayopachecoea
    @jayopachecoea 5 месяцев назад

    Gracias por la explicación 👍, es lo que buscaba para entender este tema.

  • @bantimatrix
    @bantimatrix 5 месяцев назад

    Nicely explained about basic

  • @hafizadeelarif3415
    @hafizadeelarif3415 5 месяцев назад

    In AWS Redshift cluster, what is zero ETL and how does it work, sir?

  • @Seth.Chatterley
    @Seth.Chatterley 5 месяцев назад

    Perfect video. Great walkthrough!

  • @preetybaderiya7268
    @preetybaderiya7268 5 месяцев назад

    its awesome for bignners

  • @BharathiJayaraman-m1p
    @BharathiJayaraman-m1p 5 месяцев назад

    Hi, Very Nice learning content. I am looking to create Stored Proc in Data proc! Any thoughts on it?

  • @shairy79
    @shairy79 5 месяцев назад

    I am confident now after watching this tutorial. Looking now for more advance topics tutorials.

  • @KiranSingh-t4e
    @KiranSingh-t4e 5 месяцев назад

    Great series you have created for DB 203. If possible, please create a video on Microsoft purview. thank you so much!

  • @kasper-7vfwr
    @kasper-7vfwr 5 месяцев назад

    You say airflow and dbt will run in their own container so they dont run in the same one. Where is this defined? Somewhere in the docker-compose file?

  • @OPopoola
    @OPopoola 6 месяцев назад

    Thanks. The best intro to dbt yet.

  • @NikitaLalwani-q7w
    @NikitaLalwani-q7w 6 месяцев назад

    hello that yaml file is giving error on AWS

    • @MohamedAhmed-mj2gr
      @MohamedAhmed-mj2gr 2 месяца назад

      I am also,Can you solve the error ?

    • @VishnuReddy-uf2ie
      @VishnuReddy-uf2ie 10 дней назад

      @@MohamedAhmed-mj2gr Try copying the file contents and create new one directly on AWS. worked for me !

  • @FallenJakarta
    @FallenJakarta 6 месяцев назад

    Thank you very much

  • @o0D3RMOT0o
    @o0D3RMOT0o 6 месяцев назад

    Thanks for the good tutorial, Few issues I ran into 1. I needed to point to a different bucket (I used the etl source bucket) for my redshift Temp bucket as the one you provided didnt exist in my environment. 2. I had to create a database connection in redshift to allow me to query it. I used temp user credentials using db name and admin username that was attached to the cluster. Hope this comment helps if anyone else runs into issues Thanks again :) Dermot

  • @Farisito
    @Farisito 6 месяцев назад

    thx

  • @risingstar1598
    @risingstar1598 6 месяцев назад

    Pls reply sir...from where can I get aws data analyst certificate

    • @ambhat3953
      @ambhat3953 3 месяца назад

      That Certification is deprecated. AWS Data engg certification is the most nearest option to it

  • @yourshema
    @yourshema 6 месяцев назад

    Good One! I am curious to know how does deletion of object replicates? I mean there is just one version of object in source and it is deleted (which is permanent delete)

  • @malebeauty
    @malebeauty 6 месяцев назад

    Thanks!

  • @darrienjohnson9053
    @darrienjohnson9053 6 месяцев назад

    thank you so much for this information! made my job easier to understand

  • @GagandeepSingh-mq1id
    @GagandeepSingh-mq1id 6 месяцев назад

    I am missing the part where you setup the project in GCP , can you share the timeline ?

  • @junweizhang1034
    @junweizhang1034 6 месяцев назад

    Best dbt tutorial for beginner u can find in RUclips! Well done dude!

  • @adamschlinker972
    @adamschlinker972 6 месяцев назад

    Thanks, Data Tech!

  • @abdulghanishaik
    @abdulghanishaik 6 месяцев назад

    it was really good to start, but screen is not visible clearly.