Today I watched the entire playlist without a break, you have a talent for explaining! Thank you. Also, perhaps the last two videos would be better placed at the beginning of the playlist.
Many thanks for the video ; great tutorial ! I have a question regarding Airflow though ; I keep seeing DAG file where task functions are defined directly into the DAG file. What if I want to import functions from python scripts from other Repositories ? How can I call them in my DAG file ? I plan to use Airflow to run many scripts I have deployed on various Git repositories. I don't want to recreate those already existing functions into my DAG file but just call those scripts with their functions in my Airflow repository. Any idea how can I achieve that ? Many thanks and keep up making great videos ! ;)
I think you would need to import the functions (defined in the external repositories )and decorate them with the "@task" decorator to make them tasks within your dag
Great video Marc! But why doesn't Visual Studio Code recognize the Airflow library even though I'm running my containers correctly? I've noticed that you're experiencing the same issue because the 'airflow.decorators' library is underlined in yellow. Is there a way to fix this?
Using tempfile module from python, can I create a temporay directory associated with a DAG RUN that will be automatically deleted at the end of a DAG RUN regardless if the DAG RUN was successful or failure or otherwise?
Today I watched the entire playlist without a break, you have a talent for explaining! Thank you. Also, perhaps the last two videos would be better placed at the beginning of the playlist.
woah! Thank you so much ❤️
Many thanks for the video ; great tutorial !
I have a question regarding Airflow though ; I keep seeing DAG file where task functions are defined directly into the DAG file.
What if I want to import functions from python scripts from other Repositories ? How can I call them in my DAG file ?
I plan to use Airflow to run many scripts I have deployed on various Git repositories.
I don't want to recreate those already existing functions into my DAG file but just call those scripts with their functions in my Airflow repository.
Any idea how can I achieve that ?
Many thanks and keep up making great videos ! ;)
I think you would need to import the functions (defined in the external repositories )and decorate them with the "@task" decorator to make them tasks within your dag
Great video Marc! But why doesn't Visual Studio Code recognize the Airflow library even though I'm running my containers correctly? I've noticed that you're experiencing the same issue because the 'airflow.decorators' library is underlined in yellow. Is there a way to fix this?
Yup, will make a video about that but you need to connect VS to your containers running Airflow 🥹
ruclips.net/video/fsMKV9A1B-I/видео.html I see that Marc has published such a video already :)
i have setup airflow with docker using astro cli. how to make code suggestion work in vscode?
I am facing the same issue since a week!
Using tempfile module from python, can I create a temporay directory associated with a DAG RUN that will be automatically deleted at the end of a DAG RUN regardless if the DAG RUN was successful or failure or otherwise?
Yes, two options:
- trigger rules
- setup and teardown tasks (this one better)
@@MarcLamberti can I set teardown to only run on dag failure?
Thank you the video is really help full
why the video resolution looks messed up lol
It should not 🥹
@@MarcLamberti It is actually, the first 15 seconds or so it was squashed, after that it's stretched, not a big deal but noticeable