Thank you all for your warm feedback ❤ Here is another video to create a more advanced pipeline with AWS and Snowflake: ruclips.net/video/wT67h9qDl1o/видео.html Enjoy ❤
Thank you so much. It was a helpful project to understand the basics of DAG, Airflow, Python Operator in Airflow and sharing data between tasks. Appreciate the work that you have been putting in here. You're definitely the best instructor on udemy for a reason. Cheers Mate!
This was a great tutorial - easy to conceptualize and a great example which was easy to follow. It's a great feeling to look at your data team's airflow github repo and actually understand what the DAGs are doing when you didn't know how airflow worked an hour ago!
Hi Marc!, I noticed you used the BashOperator in the example pipeline @13:16 Could you please explain why you chose BashOperator over PythonOperator in this case? Also, how do you typically decide which operator to use when designing your DAGs? Thank you again for sharing your knowledge...I really appreciate your efforts!
Merci beaucoup Marc, bon courage. Thank you sir, i really enjoyed learning while watching your video. Its the first time I discover your channel, definitely I'll be sharing it with my colleagues
Marc you are incredibly good at explaining. Perfect balance between details and conciseness! Finished this exercise succesfully at the first try! One thing I still do not understand is how can I have a task launch some external python programs that are managed in their own virtual environments by Poetry? Thanks
I'm new with airflow..... currently I have a server with jupyterhub+jupyterlab...I've installed airflow at the same server and I wanted to create this DAG from jupyterlab... but I don't have visibility of airflow modules within jupyter environmente despite of they are installed at the same server... How can i proceed?... and leads me to this question, where should I build one dag? what's your suggestion?
Great explanation! I still wonder how the PythonOperator would be able to make an instance of a python class and call a specific method of that class. Most of the videos I have found only seem to showcase the use of functions for the python_callable param. 🤔
How can I integrate those Deep learning model into spark or airflow, can you make a video about this like how we can integrate our ML or DL model into Airflow or spark for job scheduling
Thanks for lesson. I have a problem - TypeError: _choose_best_model() missing 1 required positional argument: 'ti'. Can you help?) returned values 8, 9 and 6
Marc, I stuck with an issue. I am trying to create multiple dagRun with same execution time, but getting exception. To overcome this, i tried to create it with microsecond precision, but still dagRuns are using "seconds" and truncating the microseconds. I also tried "replace_microseconds"=false, but no success. Please help or if you know any doc, please share.
Hi - I have passed this JSON {"Name" : "Jhonny"} in configuration JSON box before triggering manually. I want to print last two letters of the value which passed to the Name i.e. in this example "ny"..How do I print this in Airflow DAG..I am unable to print it
Marc, I reproduced the example you demonstrated, but I notice strange behavior: when the function fetches results from the training runs, the results are the same each time I run the DAG, so the same branch is always taken. It seems like the training function result gets cached and re-used. Any idea why?
Hey! Thanks for great videos. I am facing trouble while running a java jar file from airflow. Getting java command not found error message. P.s- tried with adding path in $PATH. Can not use docker.
I think someone already asked. Do you also need to install apache-airflow locally with pip in order to get code completion? Thanks for the great content!
It won't be python that "knows", it will be what airflow is doing behind the scenes. Since he's specified it as a python_callable, I imagine airflow will call the function and respond to it's return appropriately.
Hello, I am new at apache airflow. Your videos of airflow are awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php application Cron task. If yes, then it would be a great help for us if you make a video on it step by step like other videos.
Very useful ! But I 've got a problem: Airflow imports are not recognized in the my_dag file. I've done your video to install airflow so I am using WSL2 with VScode. I read that airflow is not recognized because Iam not connected to docker containers. If anyone have the same issue, please tell me how to solve it ...
in the video Running Airflow 2.0 you used a .yaml file but in this video you used .yml file can you explain the difference? can we created the dags in the folder that you defined in the previous video?
Thank you all for your warm feedback ❤ Here is another video to create a more advanced pipeline with AWS and Snowflake:
ruclips.net/video/wT67h9qDl1o/видео.html
Enjoy ❤
Thank you so much. It was a helpful project to understand the basics of DAG, Airflow, Python Operator in Airflow and sharing data between tasks. Appreciate the work that you have been putting in here. You're definitely the best instructor on udemy for a reason. Cheers Mate!
I can't express how grateful I am to you for sharing this content here with us on youtube.
Thank you and keep doing this excellent job.
Glad you enjoy it! :)
This was a great tutorial - easy to conceptualize and a great example which was easy to follow. It's a great feeling to look at your data team's airflow github repo and actually understand what the DAGs are doing when you didn't know how airflow worked an hour ago!
You are the best teacher I have ever seen before.
Thank you 🙏
Superb Narration about Airflow, with one video and simple example you cleared all my basic doubts. Thanks a lot.
Glad it was helpful!
Clear explanation for the beginners. Thank you!
this is very clear and insightful for me as a beginner, thank you! Can't wait to try it on my own
Thank you 🙏
amazing explanation of the first DAG creation in airflow! Thanks a lot
Wow this was such a great tutorial! Very easy to understand and can’t wait to try it myself!
You are a killer instructor! Following your tutorials feels like drinking French vanilla. Thumbs up!
love it, great video to start getting hands on airflow! please keep making more videos like these using different and more complex scenarios.
Thank you for sharing! I learned something new today! I appreciate your time!
Happy to help
Hi Marc!, I noticed you used the BashOperator in the example pipeline @13:16 Could you please explain why you chose BashOperator over PythonOperator in this case? Also, how do you typically decide which operator to use when designing your DAGs?
Thank you again for sharing your knowledge...I really appreciate your efforts!
Wow thanks man, that was a really good video. I learned a lot more than airflow.
Happy to help 🫶
Simple, To-point and well explained. 🔥🔥
Thank you 🙏
best tutorial on airflow DAG ✌
Merci beaucoup Marc, bon courage.
Thank you sir, i really enjoyed learning while watching your video. Its the first time I discover your channel, definitely I'll be sharing it with my colleagues
Marc you are incredibly good at explaining. Perfect balance between details and conciseness! Finished this exercise succesfully at the first try! One thing I still do not understand is how can I have a task launch some external python programs that are managed in their own virtual environments by Poetry? Thanks
Excellent your way of teaching is - Thank you
Amazing explanation. Fast and clear. Thank you a lot.
thanks a lot! it really help me get going with dags
Happy to help
Thanks that was amazing explanation
You’re welcome ❤️
The best explanation, kudos to you
u are amazing man. so clear !
AMAZING EXPLANATION! !!!
That explanation is really good. Kudos!
Thank you for the wonderful explanation
Thanks Marc. Very well explained.
That was both informative and enjoyable. Thank you Marc!
I'm new with airflow..... currently I have a server with jupyterhub+jupyterlab...I've installed airflow at the same server and I wanted to create this DAG from jupyterlab... but I don't have visibility of airflow modules within jupyter environmente despite of they are installed at the same server... How can i proceed?... and leads me to this question, where should I build one dag? what's your suggestion?
This was incredible.. Thank you Mark
Wonderful explanation. Thank you very much for the video!
Simple, Practical, Useful
How to implement the condition where accurate should run only when training model A,B,C all 3 are successful executed?
Great explanation! I still wonder how the PythonOperator would be able to make an instance of a python class and call a specific method of that class. Most of the videos I have found only seem to showcase the use of functions for the python_callable param. 🤔
Thank you vry much) I an working with my homework with your video
Really helpful! Thanks from Québec!
What's the difference between from airflow.models.dag import dag and from airflow import DAG?
Very comprehensible. Thank you!
Thank you! I started to understand...
awesome class tks
How can I integrate those Deep learning model into spark or airflow, can you make a video about this like how we can integrate our ML or DL model into Airflow or spark for job scheduling
really good content, thanks Marc!
Awesome explaination!!
Awesome, man. Many thanks!
awesome explanation!
How to call all snowflake stored procedures with one Task in another Python file , when corresponding Operators in declared in Main DAG File
Thank you. All simply and helpful
Great video! So helpful! Do a video on ETL airflow but loading into postgres or with sql operators
The PostgresOperator is the way 😁
Very useful tips! Thanks a lot!
Amazing work
the code runs successfully for me, but no updates happen to my airflow UI. I don't see the my_dag that we created. Can someone help me out?
Very useful ! Thank you for the sharing!
how do you run locally the airflow UI? when I use airflow standalone command it tells me: 'airflow airflow Invalid login. Please try again.'
This is very informative
How did you submit your script to Airflow? Only then you'll be able to view it in Web UI right?
Thanks for lesson. I have a problem - TypeError: _choose_best_model() missing 1 required positional argument: 'ti'. Can you help?) returned values 8, 9 and 6
Put ti=none in the parameters
@@MarcLamberti Thanks, i changed Airflow 1.0 to 2.0 and had success))
Very clear. Thank you
i have a question! How i can see result of pipeline. For example i have a function print('hello world') and i want to see it in screen
How you are able to get suggestions in your VSC without installing the Airflow dependencies?
how do i import a json config file that stores variables in another python script with airflow
very well explained.. thanks
brilliant and simple!
Awesome channel!!!
Really helpful session :)
🫶
Thank you so much for this vdo.. Really helpful
I am running airflow on port 8002. How to get my_dag in the panel?
This is great, thank you!
happy to help! :)
Hello sir ,I have created dags successfully but it is not visible at airflow web interface what should we have to do ?
Great video! TY!
I didn't find the link in description
How can we put best_accuracy on output?
Awesome explanation
Glad you liked it
Really great content!
Hi @marclamberti
I want ask as a Data Engineer, I want to regularly clean up airflow log file that more than 2 months old. Is it possible?
Marc, I stuck with an issue. I am trying to create multiple dagRun with same execution time, but getting exception. To overcome this, i tried to create it with microsecond precision, but still dagRuns are using "seconds" and truncating the microseconds. I also tried "replace_microseconds"=false, but no success. Please help or if you know any doc, please share.
Thanks brother!
Hi - I have passed this JSON {"Name" : "Jhonny"} in configuration JSON box before triggering manually. I want to print last two letters of the value which passed to the Name i.e. in this example "ny"..How do I print this in Airflow DAG..I am unable to print it
Marc, I reproduced the example you demonstrated, but I notice strange behavior: when the function fetches results from the training runs, the results are the same each time I run the DAG, so the same branch is always taken. It seems like the training function result gets cached and re-used. Any idea why?
Update: I added a call to random.seed() in the _training_model function, and it resolved the problem.
Great Tutorial
Hello, thanks for the content, but some probleme, when i run the dag , hava a error ERROR - name 'best_accuracy' is not defined
Sorry, I did not find any video in description that explain how to install Airflow to my PC. Can you help me, please ?
Maybe just find pip install apache-airflow
Great teaching skill. Thank you for the tut
I am running cmd airflow scheduler
Hey! Thanks for great videos. I am facing trouble while running a java jar file from airflow. Getting java command not found error message.
P.s- tried with adding path in $PATH. Can not use docker.
mannnnnnnnn you saved me today!!
Well, that’s great news 🫶
It will be great if you include in the tutorial how to open a file, save it and run it using airflow.
I think someone already asked. Do you also need to install apache-airflow locally with pip in order to get code completion? Thanks for the great content!
Yes
In Function, _choose_best_model return "accurate" .
How does the python/airflow know that "accurate" is not a string but a task_id for BashOperator ?
It won't be python that "knows", it will be what airflow is doing behind the scenes. Since he's specified it as a python_callable, I imagine airflow will call the function and respond to it's return appropriately.
Brilliant!
Nice instructor
very good tutorial
Hello, I am new at apache airflow. Your videos of airflow are awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php application Cron task. If yes, then it would be a great help for us if you make a video on it step by step like other videos.
Very useful ! But I 've got a problem: Airflow imports are not recognized in the my_dag file. I've done your video to install airflow so I am using WSL2 with VScode. I read that airflow is not recognized because Iam not connected to docker containers. If anyone have the same issue, please tell me how to solve it ...
I am having the same issues.
I can see the dag in the airflow UI but it never runs for me.
Hi Marc, please suggest me your Udemy coarse, as I am working in GCP composer
great content
in the video Running Airflow 2.0 you used a .yaml file but in this video you used .yml file can you explain the difference?
can we created the dags in the folder that you defined in the previous video?
There is no difference, it's still a yaml file :)
@@MarcLamberti can we run it without a docker-compose?
nop
@@MarcLamberti but i saw in one of the documentation we can do it by using airflow scheduler and airflow webserver commands
@@sourabhupadhye4915 Oh sorry. Yes you can but we docker-compose you can run everything at once without installing anything manually
Thank you for the great content