Customer Churn Data Analytics|Data Pipeline using Apache Airflow, Glue, S3, Redshift, PowerBI|Part 3

This is the part 3 of this customer churn python ETL data engineering project using Apache Airflow and different AWS services.
In this customer churn data engineering project, we will learn how to build and automate a python ETL pipeline that would use AWS glue to load data from AWS S3 bucket into an Amazon Redshift data warehouse thereafter connect PowerBi to the Redshift cluster to then visualize the data to obtain insight. AWS glue involves using AWS glue crawler to crawl the S3 bucket to infer schemas and then create a data catalogue on that. We can then also use Amazon Athena to write SQL queries on top of the data catalogue to get insight from the data. The AWS glue also helps in loading the crawled data onto the Redshift cluster. Apache airflow would be used to orchestrate and automate this process.
Apache Airflow is an open-source platform used for orchestrating and scheduling workflows of tasks and data pipelines. This project will entirely be carried out on AWS cloud platform.
In this video I will show you how to install Apache airflow from scratch and schedule your ETL pipeline.
Remember the best way to learn data engineering is by doing data engineering - Get your hands dirty!
Customer Churn Data Analytics|Data Pipeline using Apache Airflow, Glue, S3, Redshift, PowerBI|Part 1 • Customer Churn Data An...
Customer Churn Data Analytics|Data Pipeline using Apache Airflow, Glue, S3, Redshift, PowerBI|Part 2 • Customer Churn Data An...
If you have any questions or comments, please leave them in the comment section below.
Please don’t forget to LIKE, SHARE, COMMENT and SUBSCRIBE to our channel for more AWESOME videos.
*Books I recommend*
1. Grit: The Power of Passion and Perseverance amzn.to/3EZKSgb
2. Think and Grow Rich!: The Original Version, Restored and Revised: amzn.to/3Q2K68s
3. The Book on Rental Property Investing: How to Create Wealth With Intelligent Buy and Hold Real Estate Investing: amzn.to/3LLpXRy
4. How to Invest in Real Estate: The Ultimate Beginner's Guide to Getting Started: amzn.to/48RbuOb
5. Introducing Python: Modern Computing in Simple Packages amzn.to/3Q4driR
6. Python for Data Analysis: Data Wrangling with pandas, NumPy, and Jupyter 3rd Edition: amzn.to/3rGF73G
**************** Commands used in this video ****************
sudo apt update
sudo apt install python3-pip
sudo apt install python3.10-venv
python3 -m venv customer_churn_youtube_venv
source customer_churn_youtube_venv/bin/activate
sudo pip install apache-airflow
pip install apache-airflow-providers-amazon
airflow standalone
CREATE TABLE IF NOT EXISTS customer_churn(
CustomerID VARCHAR(255),
City VARCHAR(255),
Zip_Code INTEGER,
Gender VARCHAR(255),
Senior_Citizen VARCHAR(255),
Partner VARCHAR(255),
Dependents VARCHAR(255),
Tenure_Months INTEGER,
Phone_Service VARCHAR(255),
Multiple_Lines VARCHAR(255),
Internet_Service VARCHAR(255),
Online_Security VARCHAR(255),
Online_Backup VARCHAR(255),
Device_Protection VARCHAR(255),
Tech_Support VARCHAR(255),
Streaming_TV VARCHAR(255),
Streaming_Movies VARCHAR(255),
Contract VARCHAR(255),
Paperless_Billing VARCHAR(255),
Payment_Method VARCHAR(255),
monthly_charges FLOAT,
Total_Charges FLOAT,
Churn_Label VARCHAR(255),
Churn_Value INTEGER,
Churn_Score INTEGER,
Churn_Reason TEXT
)
**************** USEFUL LINKS ****************
How to remotely SSH (connect) Visual Studio Code to AWS EC2: • How to remotely SSH (c...
PostgreSQL Playlist: • Tutorial 1 - What is D...
Github Repo: github.com/Yem...
www.kaggle.com...
Apache Airflow Playlist • How to build and autom...
How to build and automate a python ETL pipeline with airflow on AWS EC2 | Data Engineering Project • How to build and autom...
airflow.apache...
airflow.apache...
registry.astro...
Download PowerBI www.microsoft....
DISCLAIMER: This video and description has affiliate links. This means when you buy through one of these links, we will receive a small commission and this is at no cost to you. This will help support us to continue making awesome and valuable contents for you.