Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
HTML-код
- Опубликовано: 17 окт 2024
- Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/...
In this project, you will learn:
✅ How to set up an Airflow environment with the Astro CLI
✅ How to set up and configure Snowflake with Airflow
✅ How to load data from an HTTP endpoint into a Snowflake table
✅ How to use python virtual environments to avoid dependency conflicts
✅ How to run data quality checks with Soda and the ExternalPythonOperator
Enjoy ❤️ - Наука
I just added this video to my weekend RUclips playlist, but thanks, Mark, for consistently creating great tutorials and sharing tips for data people in simple human language.
Thank you ❤️
Kindly assist; Still facing connection issues: getting error snowflake.connector.errors.OperationalError: 250001: 250001: Could not connect to Snowflake backend after 2 attempt(s).Aborting,
Great video.
This complete environment that you used can I to use in prod? (astro, dbt soda, etc)
And eveything is free?
Yes except snowflake
@MarcLamberti
can we do soda integration wthout storing Snowflake password in code?
Do we need Docker Desktop to installed in the machine?
Yes
I have a problem with @task.external_python decorator. It seems that it also requires airflow package in soda_venv virtual environment which is kinda odd..
You can find the code in the link in description from this video ruclips.net/video/DzxtCxi4YaA/видео.html :)
@@MarcLamberti Particularly, I am getting an error "ModuleNotFoundError: No module named 'airflow'" and "ModuleNotFoundError: No module named 'pendulum'" when running "astro@f867522c2cb4:/usr/local/airflow$ airflow tasks test movie check_movie" (the same error is obtained also if I step into the virtual environment using "astro@f867522c2cb4:/usr/local/airflow$ source soda_venv/bin/activate")
How to enable Test connection in Airflow.
Is this tutorial anywhere on Git?
Great video Marc, Very well put together. 👏👏👏👏👏👏
PS: It would been nice if you add the link to the source movie file in the comment or description for easier follow up.
Thank you so much! will do
*PromoSM*