Monitor Apache Airflow with StatsD | Prometheus | Grafan | part1

#apacheairflow #airflowforbeginners #airflowmetrics #statsd #prometheus #grafana
In this video, you'll learn about how you can monitor airflow cluster by taking advantages of useful airflow statsd metrics. As part of single docker-compsoe file, we will spin up following services:
1. airflow-init: initializes airflow database schema and creates admin user
2. airflow-scheduler: runs airflow scheduler in LocalExecutor
3. airflow-webserver: runs airflow webserver exposing on port 8081
4. mysql: runs mysql database for airflow metadata
5. statsd-exporter: service that converts airflow statsd metrics to Prometheus metrics
6. Prometheus: runs Prometheus service that scraps airflow metrics from statsd-exporter
7. Grafana: runs Grafana instance where our final dashboards will be available.
---------------------------------------------------
👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.
⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.
---------------------------------------------------
Timeline:
00:00 Intro
00:56 Monitoring data flow
01:41 Airflow metrics configs
02:01 Write docker-compose file
03:15 writing statsd-exporter service
04:02 statsd config mapping
06:01 test metrics
08:42 outro
-----------------------------------------------
Get complete code from here;
github.com/max...
Grafana and StatsD configs
github.com/dat...
Statsd python client
github.com/jso...
Watch the previous video here:
Run Airflow locally via Docker
• Build your first pipel...
Learn Something New Below:
maxcotec.com/b...
maxcotec.com/b...