Beautiful explanation and a great project to get me started! Many thanks vik!! One thing to add from my experience: I installed airflow on my Mac M1 and it was working fine but I couldn't run any of the tasks we performed here (not even in the get_episodes task).. to solve that I made an EC2 instance and with some tweaks everything ran :D
Debug tip #1: If you run into error "conn_id isn't defined", then use the Airflow browser interface to instead create the connection. Select Admin --> Connections --> + Debug tip #2: If your Airflow runs fail, try logging out of the Airflow UI and restarting the Airflow server by pressing Ctrl + C and then airflow standalone.
if you are facing an issue with creating database, that your dag is running and not completing then put this line after importing the packages os.environ['NO_PROXY'] = '*' , it will work then for sure
Hello Vikas Thanks for such a great tutorial everting you made smooth like butter thanks for that ,just one question whenever we made new DAG ( we will have to add docker-compose-CeleryExecutor, docker-compose-LocalExecutor, and Config for that particular DAG )
I am stuck at 14:50 where you try to run the task in airflow. You simply switched the screen from pycharm terminal to git master terminal without any explanation, and I am unable to connect sqlite to pycharm terminal, neither could I establish connection with airflow. Please guide me through this
So where is the dependency chain where you set the actual task flow? I would have expected something like task1 >> Task2, etc... at the bottom of the Dag.
At 33:48, how did we get the 'Done loading. Loaded a total of 0 rows'. We haven't used this text in our code anywhere. Is this the work ok hook.insert_rows
hi everyone. I have configured airflow with docker in a folder which include docker yaml file. Now I want to use airflow in another folder, so How can i do it without docker yaml file ? must I configure again in that folder ?
really liked this tutorial. The download episodes task is freezing on me. The task is "running" but it appears to be getting held up and has not actually downloaded the first episode for some reason
That's strange - can you access the podcast site in your browser? It may be blocking you for some reason. It's also possible that the airflow task executor isn't running properly.
I ran into the same issue except on the very first task "get_episodes" nothing happens and eventually times out. tested the script in python console and it returned the list of episodes just fine. @@Dataquestio
I need help. For the final task it showed audio_path no such file or directory. So i used 'os.makedirs(audio_path, exist_ok=True)'. The dag was a success. But i couldnt find any files in my episodes folder. Please help
def download_episodes(episodes): for episode in episodes: filename = f"{episode['link'].split('/')[-1]}.mp3" audio_path = os.path.join("episodes", filename) if not os.path.exists("episodes"): os.makedirs("episodes") if not os.path.exists(audio_path): print(f"Downloading {filename}") audio = requests.get(episode["enclosure"]["@url"]) with open(audio_path, "wb+") as f: f.write(audio.content)
it happens because your airflow might not be directing to correct folder so it will create the folder somewhere ese where it is pointing, you can then search the folder to find out where it is
Add these lines to your airflow.cfg config file : min_file_process_interval = 0 dag_dir_list_interval = 30 and then the dags will be updated automatically. P.S : Be sure that your DAG is in the dags file.
Awesome tutorial! Just had some confusion on the transform and loading function, particularly this code: stored = hook.get_pandas_df('SELECT * FROM episodes;') I thought you were querying from the episode list that was returned from the extract function, but suddenly realized that it was also the same as the Table name in the database lol.
Hi Park - that code is selecting from the sqlite3 database that we create. It's making sure the podcast hasn't been stored in the database yet (if it is, we don't need to store it again).
![Kodak Black - Sharp Vibes [Official Music Video]](http://i.ytimg.com/vi/oEV0OoLJeIs/mqdefault.jpg)
I think you’re the best data science lecturer so far. Keep going thanks for your hard work
Thanks :) -Vik
Beautiful explanation and a great project to get me started! Many thanks vik!!
One thing to add from my experience: I installed airflow on my Mac M1 and it was working fine but I couldn't run any of the tasks we performed here (not even in the get_episodes task).. to solve that I made an EC2 instance and with some tweaks everything ran :D
It was very useful. Thank you. It will be really helpful if you cover Apache Hadoop, Spark, MLFlow, Flink, Flume, Pig, Hive etc. Thank you
Thank you very much for this amazing tutorial :D
This video was a great resource. Thanks for the tutelage and your take on it.
What a amazing tutorial ! Thank you a lot
Can you make more advanced Apache Airflow tutorials too?
Debug tip #1: If you run into error "conn_id isn't defined", then use the Airflow browser interface to instead create the connection. Select Admin --> Connections --> +
Debug tip #2: If your Airflow runs fail, try logging out of the Airflow UI and restarting the Airflow server by pressing Ctrl + C and then airflow standalone.
this is very educational thanks a lot, i was wondering if you would be making a video of the automatic transcriptions
Yes, I will be doing a webinar for this tomorrow, and the video should be live next week on RUclips. -Vik
👍👍👍excellent tuto. Thks a lot
nice one @Vik
if you are facing an issue with creating database, that your dag is running and not completing then put this line after importing the packages os.environ['NO_PROXY'] = '*' , it will work then for sure
thanks you for your tutorials, let me know about your airflow version on your tutorial to practice.
Hello Vikas Thanks for such a great tutorial everting you made smooth like butter thanks for that ,just one question whenever we made new DAG ( we will have to add docker-compose-CeleryExecutor, docker-compose-LocalExecutor, and Config for that particular DAG )
I am stuck at 14:50 where you try to run the task in airflow. You simply switched the screen from pycharm terminal to git master terminal without any explanation, and I am unable to connect sqlite to pycharm terminal, neither could I establish connection with airflow. Please guide me through this
whenever i run airflow standalone (or even airflow webserver) i get the ModuleNotFound error for pwd. Im running it on a windows, how do i solve this?
So where is the dependency chain where you set the actual task flow? I would have expected something like task1 >> Task2, etc... at the bottom of the Dag.
At 33:48, how did we get the 'Done loading. Loaded a total of 0 rows'. We haven't used this text in our code anywhere. Is this the work ok hook.insert_rows
hi everyone. I have configured airflow with docker in a folder which include docker yaml file. Now I want to use airflow in another folder, so How can i do it without docker yaml file ? must I configure again in that folder ?
really liked this tutorial. The download episodes task is freezing on me. The task is "running" but it appears to be getting held up and has not actually downloaded the first episode for some reason
That's strange - can you access the podcast site in your browser? It may be blocking you for some reason. It's also possible that the airflow task executor isn't running properly.
I ran into the same issue except on the very first task "get_episodes" nothing happens and eventually times out. tested the script in python console and it returned the list of episodes just fine.
@@Dataquestio
Fix for me was to start the airflow server as the root user; "sudo airflow standalone"
When I change the 'dags_folder' path, everything breaks in airflow. What could be the reason
you're first bit on seting the environment variable didn't work
hello. I did everything as it is but it fails and no logs are visible
My download_episodes task succeeds but I cannot see the mp3 files
I need help. For the final task it showed audio_path no such file or directory. So i used 'os.makedirs(audio_path, exist_ok=True)'. The dag was a success. But i couldnt find any files in my episodes folder. Please help
def download_episodes(episodes):
for episode in episodes:
filename = f"{episode['link'].split('/')[-1]}.mp3"
audio_path = os.path.join("episodes", filename)
if not os.path.exists("episodes"):
os.makedirs("episodes")
if not os.path.exists(audio_path):
print(f"Downloading {filename}")
audio = requests.get(episode["enclosure"]["@url"])
with open(audio_path, "wb+") as f:
f.write(audio.content)
it will create the episodes folder if it is not created
it happens because your airflow might not be directing to correct folder so it will create the folder somewhere ese where it is pointing, you can then search the folder to find out where it is
Please kindly help, when I write my DAG on vscode, it doesn't show on the WEB UI and I keep getting this DAGNOTFOUND Error
Add these lines to your airflow.cfg config file :
min_file_process_interval = 0
dag_dir_list_interval = 30
and then the dags will be updated automatically.
P.S : Be sure that your DAG is in the dags file.
how did you change your terminal to git:(master)
I use a terminal called zsh. There is a plugin for zsh that can show you your git branch.
@@Dataquestio thx man
How did he get the web page metadata in Xml?
Look at the link in the video, type that link in a Chromium browser and it should appear. Wala ..
@@HJesse88 it didn't appear for me
Awesome tutorial!
Just had some confusion on the transform and loading function, particularly this code:
stored = hook.get_pandas_df('SELECT * FROM episodes;')
I thought you were querying from the episode list that was returned from the extract function, but suddenly realized that it was also the same as the Table name in the database lol.
Hi Park - that code is selecting from the sqlite3 database that we create. It's making sure the podcast hasn't been stored in the database yet (if it is, we don't need to store it again).