Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.
Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?
@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?
i am currently have an issue with subtasks in apache airflow. EXAMPLE: let say there are 3 subtasks (in parallerl) which are already completed and now two new tasks are added.. HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running) Attempts: from GUI perspective we have option called "queue up new tasks" that does the job. I am coming from the script point of view , about how to activate this button/option Could you please provide me with a hint/idea/suggestions (literally anything )
I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml
how to create dags or jobs using this latest ui i dont see any option to do so
setup is just the first task, and teardown is the last task with a "all_done" trigger rule, what do these new labels bring ?
Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown
setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.
@@MarcLamberti I'll read it right away, thank you Marc
Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?
Will do 🫶
good point, indeed
Where do i find task logs ?
On the UI
@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?
i am currently have an issue with subtasks in apache airflow.
EXAMPLE:
let say there are 3 subtasks (in parallerl) which are already completed
and now two new tasks are added..
HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running)
Attempts:
from GUI perspective we have option called "queue up new tasks" that does the job.
I am coming from the script point of view , about how to activate this button/option
Could you please provide me with a hint/idea/suggestions (literally anything )
I suggest you open a stack overflow topic, I'm not convinced this comment has anything to do with "what's new in Airflow 2.7"
I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml
Will do
Way to less likes! Your videos are top! Thanks
Please do make a video about setup and teardown
Will do 🤓