Airflow Vs. Prefect: Full Breakdown!
HTML-код
- Опубликовано: 15 мар 2023
- Today we're taking a deep dive into two of the most popular data orchestration tools on the market today! We'll look at a few main areas:
- Workflow Creation
- Dynamic Workflows
- Scalability
- Monitoring/UI
Hope you learn something, and drop your own opinion in the comments below!
bro, Prefect uses function relationships to create its order of operations. it can absolutely by dynamic. that YAML file is used for deployments...
It can be dynamic but requires a lot of extra steps and doesn't have the same amount of customizability as Airflow IMO, but if you want to come on and debate I would love to have ya!
Amazing content, as usual. I work with airflow daily, which I think it is a amazing tool. Trying to learn a new orchestration tool to be used when need on some freelancers projects that I'm aiming to take. I choose Airflow as my main orchestration DE tool and if needed use Prefect. Tried Dagster but didn't liked the UI and documentation.
Thanks, appreciate you tuning in!! Airflow + Prefect is a great combo, have some of my hedge fund friends doing that so they can train models on terabytes of data on desk!
Thanks for the breakdown. Very informative. I would say Airflow vs Dragster will help too.
Just dropped the video on that, hope you enjoy!
Hey, I would love to have a conversation with you about these two tools and their differences to better understand a couple of the points that you made on the video. Let me know if you’d be open to that. Great video by the way. Cheers!
Definitely, just hit me up on my email, gyatesofficial@gmail.com
I don't understand the criticism about the absence of dynamic DAG in Prefect. Of course you can dynamically launch tasks through loops and what-not. A Prefect Flow is not a hard-coded script without parameters and branches.
Also, you focused on the yaml file that is used for deployment, but you didn't show the actual python code, which is very pythonic and easy to handle for a newcomer. And that is, I think, a big strength of Prefect.
I guess my point on Dynamic DAG's is that while you can accomplish it through loops, that isn't really an OOTB feature, but rather one you need to build yourself. You're right, I should have shown the python scripts, I think my point with that section is that having to manage all the separate python files in conjunction with the .yaml file can be difficult to manage at large scale.
Dude - this was fantastic, subscribed!
I wish I found your channel before passing the Astronomer exam, you're fun to watch!
It's actually funny we have the exact same # subs atm - wishing you the best!
Hahaha wow that's a crazy coincidence! Thanks man appreciate it, toss you a sub as well to keep us at parity ;) Content looks great as well!
How about dagster?
If you're interested I'll make the video!
Just dropped the dagster video, check it out!
Did you just call yourself Data Orchestration GURU? 😂
Hahahaha yeah that was the old name before I realized the negative connotations around Guru's!