Hello, you are the best Data Engineering Instructor here on RUclips. I want to continue learning on your end to end project unfortunately I am having a problemin in initiating airflow. scheduler | [2024-05-05 04:41:04 +0000] [3014] [INFO] Booting worker with pid: 3014 scheduler | [2024-05-05T04:41:04.168+0000] {settings.py:60} INFO - Configured default timezone UTC scheduler | [2024-05-05 04:41:04 +0000] [3016] [INFO] Booting worker with pid: 3016 scheduler | [2024-05-05T04:41:04.405+0000] {manager.py:393} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2) when using sqlite. So we set parallelism to 1. this is the error came from my terminal. Hope you can assist me sir. Thank you and more power
You are the best instructor out there on RUclips, thank ou for explaining everything. I would like for you to do another data engineering with spark and EMR. Finally, maybe do some video on data analysis and data processing. Finally, please keep going and let us know how we can support you to continue on this amazing path!!!
Thanks so much for your kind words. Means a lot to me and yes we should be having a data engineering project on spark and EMR soon. There are different ways by which you can support my channel. You can decide to support me by being a member of this channel. Please listen to this short video where I explained the JOIN membership button. ruclips.net/video/I-72fMtbEbE/видео.html . You could also support me with a "Super Thanks" which you can see at the button of every of my videos. Another way to support me is by Liking my videos and sharing with others so that they can also benefit from these videos. Thanks so much!
Thanks so much for your kind words. Here is the project EMR ruclips.net/video/k3lIQZzsaWY/видео.htmlfeature=shared and ruclips.net/video/PeaLln90YXg/видео.htmlfeature=shared
Thank you so much for the video, I am getting error "ModuleNotFoundError: No module named 'boto3'" even I installed boto3 following everything as you instructed. Please share your thoughts on this?
Hi there, first of all I just want to appreciate you for helping out in self learning journey of people like us. I closely follow your projects and implement it whenever I can. For this tutorial as well, I followed everything as you instructed but at last to load data to s3 failed for some reason. My raw data bucket was empty but the transformed data of 867MB was successfully saved. Can you point me towards the right direction here? Also, I'm not using root account and my user has Admin level access. ~Thank you for your time :)
You don't have to wait 5 min for DAGs to be reloaded. Run this in shell > airflow dags reserialize No need to stop the airflow, open another shell and activate venv.
![[4K] Watch SpaceX Catch A Starship Rocket From Space!!! #IFT5](http://i.ytimg.com/vi/pIKI7y3DTXk/mqdefault.jpg)
God bless you and your beautiful soul for taking the time and effort to make this and make it available for free 🙏
You are so welcome
Hello, you are the best Data Engineering Instructor here on RUclips. I want to continue learning on your end to end project unfortunately I am having a problemin in initiating airflow.
scheduler | [2024-05-05 04:41:04 +0000] [3014] [INFO] Booting worker with pid: 3014
scheduler | [2024-05-05T04:41:04.168+0000] {settings.py:60} INFO - Configured default timezone UTC
scheduler | [2024-05-05 04:41:04 +0000] [3016] [INFO] Booting worker with pid: 3016
scheduler | [2024-05-05T04:41:04.405+0000] {manager.py:393} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2) when using sqlite. So we set parallelism to 1.
this is the error came from my terminal. Hope you can assist me sir. Thank you and more power
Consider Switching to a More Robust Database
You are the best instructor out there on RUclips, thank ou for explaining everything. I would like for you to do another data engineering with spark and EMR. Finally, maybe do some video on data analysis and data processing. Finally, please keep going and let us know how we can support you to continue on this amazing path!!!
Thanks so much for your kind words. Means a lot to me and yes we should be having a data engineering project on spark and EMR soon. There are different ways by which you can support my channel. You can decide to support me by being a member of this channel. Please listen to this short video where I explained the JOIN membership button. ruclips.net/video/I-72fMtbEbE/видео.html . You could also support me with a "Super Thanks" which you can see at the button of every of my videos. Another way to support me is by Liking my videos and sharing with others so that they can also benefit from these videos. Thanks so much!
very very nice detailed explanation which is very helpfull to understand topic clearly...Thanku so much please keep posting videos sir
You are welcome and thanks for your comment. It means alot to me.
Great Content! I will definitely follow along! Quick question, how much will the EC2 t2 xlarge instance cost be for this whole project?
thank you so much for this video. I use a mac and i wanted to know if i could select macos instead of ubuntu and why ubuntu is better
This a great project, you are doing a great job. I am waiting for this same project wiht EMR!!!
Thanks so much for your kind words. Here is the project EMR ruclips.net/video/k3lIQZzsaWY/видео.htmlfeature=shared and ruclips.net/video/PeaLln90YXg/видео.htmlfeature=shared
Great video, can't wait for part 2
Glad you enjoyed it. Thank you! See you soon in part 2.
Awesome explanation
Glad you liked it
How did you made connection with airflow? Is it HTTP?
Thank you so much for the video, I am getting error "ModuleNotFoundError: No module named 'boto3'" even I installed boto3 following everything as you instructed. Please share your thoughts on this?
Make sure to activate your virtual environment before doing "pip install boto3".
Hi there, first of all I just want to appreciate you for helping out in self learning journey of people like us. I closely follow your projects and implement it whenever I can.
For this tutorial as well, I followed everything as you instructed but at last to load data to s3 failed for some reason. My raw data bucket was empty but the transformed data of 867MB was successfully saved. Can you point me towards the right direction here? Also, I'm not using root account and my user has Admin level access.
~Thank you for your time :)
Did you remember to refresh your raw data s3 bucket so as to check if the file was saved there?
very very very helpful!!! thank you so much!!
Glad it was helpful!
Great, let's do this!
Awesome. Thanks!
You don't have to wait 5 min for DAGs to be reloaded. Run this in shell > airflow dags reserialize
No need to stop the airflow, open another shell and activate venv.
Amazing!!
Thank you!
great project!
Thanks so much.
Tip: do not use ctrl+c to copy the airflow password as this will also cause the server to shut down 😂
True...Thanks for pointing it out.
I like your accent
Thanks! 😁
dag is not showing
dag is not showing
It might take about 5mins before dag shows. Otherwise, you might be having some errors.
> airflow dags reserialize
This reloads dags immediately