The New Way of Scheduling DAGs in Airflow with Datasets

Поделиться
HTML-код
  • Опубликовано: 5 фев 2025
  • Airflow Datasets bring a new way of scheduling your data pipelines
    👍 Smash the like button to become an Airflow Super Hero!
    ❤️ Subscribe to my channel to become a master of Airflow
    🏆 BECOME A PRO: www.udemy.com/...
    🚨 My Patreon: / marclamberti
    Ready?
    Let's go!

Комментарии • 34

  • @trench6118
    @trench6118 2 года назад +4

    Airflow has been on fire lately - I love TaskFlow API and dynamic task mapping. Data aware scheduling came out at a perfect time and simplified a real problem for me

    • @MarcLamberti
      @MarcLamberti  2 года назад +1

      Other great features are coming. Stay tuned ;)

  • @richie.edwards
    @richie.edwards 2 года назад +2

    I started working more with Airflow at my job and your videos have been very helpful when I want to switch up learning format and not look through docs to get exposed to concept.

  • @ЕвгенийПрочан
    @ЕвгенийПрочан Год назад +1

    This is awesome. No more such ugly Triggers, Sensors and etc. Thx for explanation Marc!

  • @alina_do_min
    @alina_do_min 5 дней назад

    Thank you very much for the explanation! It turned out very cool!

  • @brunosompreee
    @brunosompreee Год назад +1

    Great content as always Marc!

  • @eduardofarias87
    @eduardofarias87 Месяц назад

    I imagine this solution in a different way. I thought it would be just putting a file or updating a file in the dataset directory (or the file) without the need for a "producer" dag. I have an external system and not a DAG that unloads the data into a directory. For me, the "TriggerDagRunOperator" already fulfills exactly the same role as the Datasets. Help me understand if that's not it. Thanks for the content Marc!

  • @RobsonLanaNarvy
    @RobsonLanaNarvy Год назад +1

    Nice demonstration, I will test a MySQL as dataset to explore this feature

  • @practicalgcp2780
    @practicalgcp2780 2 года назад +2

    Amazing video Marc! This is a truly amazing feature! Although, one thing I couldn't seem to find is a way to pass some parameters to the consumer DAG. Is there a way to access the context of what triggered the DAG? Or the extra params can be passed in the Dataset? This could be useful metadata such as the latest timestamp of some data been updated which can be useful to the downstream processes when triggered. Thank you!

  • @minnieshi2934
    @minnieshi2934 Год назад +1

    Same thing, great /not direct comment about:
    If the producer DAG ‘s task had defined the outlet, but does not really access the file/folder. Or has nothing to do with the content of the URI in the task logic, what would happen? the consumer DAG still runs.
    So it is really just using the URI as a link between the two DAGs.

    • @VallabhGhodkeB
      @VallabhGhodkeB Год назад

      Yeah exactly it is just the URI that acts as a bridge. It does not actually point to anything

  • @askmuhsin
    @askmuhsin 2 года назад +1

    Hi Marc, This is indeed a truly amazing feature.
    Just wondering if there is always going to be an instance of consumer DAG triggered for every file (URI) change.
    ie.. in case the consumer DAG is running while the producer DAG has created a new file change, will the new change cause a new coumser DAG instance to run on the new data (ie.. while the previous conumser instance is still running). if that makes sense.
    as always thank you for the content.

  • @rohithspal
    @rohithspal 2 года назад +3

    A very utilitarian feature!..
    Isn't "task aware scheduling" a more appropriate name for this feature? Since there is no real interaction with data .

    • @MarcLamberti
      @MarcLamberti  2 года назад +2

      I think there will be real interaction with data at some point 😉

  • @tiankun4450
    @tiankun4450 Год назад +1

    can i use template var (like ds_nodash) in Dataset uri ?

  • @Jeoffrey54
    @Jeoffrey54 2 года назад

    Amazing 👀

  • @davideairaghi6763
    @davideairaghi6763 2 года назад +1

    Hi Marc, datasets looks like to be very useful but how they can be used to trigger a dag based on a SQL database update? Is there any example of it? Thanks in advance

  • @Empusas1
    @Empusas1 Год назад

    You mentioned that the consumer dag triggered by the dataset is alway run when the producer dag did run successfully, not when the dataset has changed. Let`s say the producer has a compare task, and only changes the dataset if necessary. In that case the consumer would always run anyway. Any way to solve that?

    • @Alex-4ch
      @Alex-4ch 4 месяца назад

      What if pass extras for the consumer with the flag to act or no

  • @bettatheexplorer1480
    @bettatheexplorer1480 2 года назад

    is this only available on airflow >= 2.4 ?

  • @alfahatasi
    @alfahatasi 9 месяцев назад

    How to extract table from postgre database instead of txt file as dataset. Is there an example video for this?

  • @RalfredoSauce
    @RalfredoSauce 2 года назад

    how do we trigger it off an SQL table update rather than a file? He mentions its possible but I can't seem to find documentation for it anywhere

  • @ady3949
    @ady3949 Год назад

    Hi Marc, This is indeed amazing feature.
    I try to use dataset scheduling, but when job finished/or failed, it is didn't trigger my on_success_callback/my_failure_callback. While using normal scheduling (ex: @hourly, etc), it trigger my on_success_callback/my_failure_callback. Is there any config that I missed? or is it a bug?

  • @lifeofindians1695
    @lifeofindians1695 2 года назад

    Hi Marc,
    I am watching your airflow architecture video
    In single node Executor update the metastore
    In multi architecture executor put data in queue
    So who will update the metastore in multinode after job is done
    Queue or executor

  • @minnieshi2934
    @minnieshi2934 Год назад

    Very good to comment that external system updates the dataset file will NOT make the consumer DAG to run.

    • @MarcLamberti
      @MarcLamberti  Год назад

      Not yet. But it will be possible very soon

  • @imosolar
    @imosolar 2 года назад

    please update the udemy with dataset