Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?
Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.
I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml
i am currently have an issue with subtasks in apache airflow. EXAMPLE: let say there are 3 subtasks (in parallerl) which are already completed and now two new tasks are added.. HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running) Attempts: from GUI perspective we have option called "queue up new tasks" that does the job. I am coming from the script point of view , about how to activate this button/option Could you please provide me with a hint/idea/suggestions (literally anything )
@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?
Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?
Will do 🫶
good point, indeed
setup is just the first task, and teardown is the last task with a "all_done" trigger rule, what do these new labels bring ?
Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown
setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.
@@MarcLamberti I'll read it right away, thank you Marc
how to create dags or jobs using this latest ui i dont see any option to do so
I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml
Will do
i am currently have an issue with subtasks in apache airflow.
EXAMPLE:
let say there are 3 subtasks (in parallerl) which are already completed
and now two new tasks are added..
HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running)
Attempts:
from GUI perspective we have option called "queue up new tasks" that does the job.
I am coming from the script point of view , about how to activate this button/option
Could you please provide me with a hint/idea/suggestions (literally anything )
I suggest you open a stack overflow topic, I'm not convinced this comment has anything to do with "what's new in Airflow 2.7"
Way to less likes! Your videos are top! Thanks
Please do make a video about setup and teardown
Will do 🤓
Where do i find task logs ?
On the UI
@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?