Data-Aware Scheduling in Airflow 2.4

The focus of today's content is to demonstrate the utilisation of the Data-Aware Scheduling functionality incorporated into Airflow 2.4, for the purpose of resolving large-scale data coordination issues between multiple functional teams. If you are part of a sizeable organisation that currently uses Airflow, this could prove especially advantageous. It's worth noting that Google recently released Airflow 2.4.3 on Cloud Composer, so you can readily upgrade and start leveraging this feature.
A few useful things to follow up with
- Really good video on how this feature works from Marc: • The New Way of Schedul...
- The code repo: github.com/roc...
- The composer versions page: cloud.google.c...

[@lukasleitner8662]

Awesome video! It was really helpful and well done.
Thanks!

[@practicalgcp2780]

Glad it helped Lukas!

[@ap2394]

Thanks for detailed video. Can we have scheduling at task level ? Eg : if have 2 task in downstream DAG and both are different on different dataset. Can I control the schedule at task level ?

[@practicalgcp2780]

Just realised I never replied to this one, my apologies. I am not sure that is the right way to think about how this works. Regardless which task it is, or which dag, it’s about listening to a change event from something got triggered in the upstream dataset, then react to that event.
As long as you design dags in a way it is the right behaviour to trigger a dag, based on a change event, then it will work.