How to build and automate a python ETL pipeline with airflow on AWS EC2 | Data Engineering Project

Поделиться
HTML-код
  • Опубликовано: 17 окт 2024

Комментарии • 278

  • @Zelinity
    @Zelinity 9 месяцев назад +11

    This video has the signature of a master teacher. You introduced key concepts in a way that is simple to understand. Thank you for starting from level 1 without any assumptions of what we viewers/learners bring to the subject.

    • @tuplespectra
      @tuplespectra  9 месяцев назад

      Thanks so much for this comment. It really means a lot to me. I'm glad you found it valuable.

    • @prabhatgupta6415
      @prabhatgupta6415 9 месяцев назад

      bring more on databricks as well end to end please. @@tuplespectra

  • @hq1n
    @hq1n 8 месяцев назад +5

    idk why you are not at least as hype as Zach Wilson. thank you very much for giving out high quality content for free!

    • @tuplespectra
      @tuplespectra  8 месяцев назад

      Thanks so much for your comment. I'm glad you found the video helpful.

  • @vaibhavpawar3403
    @vaibhavpawar3403 10 месяцев назад +5

    Very detailed and basics tutorial with actual hands-on recorded. No PPT's simply a basic teaching which is very helpful for data engineer.

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      Glad it was helpful! Thanks so much for your comment.

  • @toml2951
    @toml2951 Месяц назад +1

    Thank you, for this! Love the enthusiasm, exuberance and energy brother. Usually, most tutorials are dull, monotonous and riddled with confusing, and abstract terms and acronyms. You really take the time to distill things in plain speak!

  • @rex2758
    @rex2758 Год назад +5

    Thanks for taking the time to talk this video out!

  • @hellowillow_YT
    @hellowillow_YT 10 месяцев назад +1

    I've been looking for an ETL project videos that I can follow to learn basic data engineering stuff and finally I found your video! Thank you for this!

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      Awesome, thank you! I'm glad it was helpful. Please go ahead to explore other videos to take your skill to the next level.

  • @nickernara
    @nickernara 8 месяцев назад

    i was initially puzzled or worried if i can grasp all but thanks for this video. this helps to dive in to code with airflow

  • @nnamdinwafor
    @nnamdinwafor 7 месяцев назад

    After watching this video, I knew I had to thank you for this truly awesome video. I have learnt more from this video than from many others out there. You are amazing.

    • @tuplespectra
      @tuplespectra  7 месяцев назад

      Great to hear! Thanks so much for this comment. It means a lot to me.

  • @murilloalves14
    @murilloalves14 Год назад +8

    This was just what I was looking for! Now it's time to apply it on my own projects. Keep the good work! Big thank you from Brazil!

    • @tuplespectra
      @tuplespectra  Год назад

      Thank you! I'm glad you like the video. Always a good idea to apply the learning in another project. Goodluck!

  • @kerryw6361
    @kerryw6361 11 месяцев назад +1

    This is really good info! Thank you! One possible area to further advance this video is to upgrade the final task by loading data to an actual database (PostgreSQL for example).

    • @tuplespectra
      @tuplespectra  11 месяцев назад

      Thanks so much for your comment. Means a lot to me. I agree that we can add a final task to load the data in a database. I have done that in other videos where we loaded to PostgreSQL and in another video we loaded to AWS redshift and yet in another video we loaded to snowflake. Please see my airflow playlist for several airflow projects to explore. Thanks so much. ruclips.net/video/DKsf88oCPWA/видео.html

  • @gyungyoonpark
    @gyungyoonpark 8 месяцев назад

    this video is so clear and helpful. there are many airflow courses, but this video goes beyond and helps you "practice" airflow. hats off to the master and look forward to more awesome videos!!!

  • @mahmoudferhat8031
    @mahmoudferhat8031 7 месяцев назад +1

    simple and clearly explained, THANKS !!

    • @tuplespectra
      @tuplespectra  6 месяцев назад +1

      Thanks for your comment. I'm glad you found the video helpful.

  • @mdobaidullahal-faruk3457
    @mdobaidullahal-faruk3457 4 месяца назад +1

    Thank you very much for making the concepts so easy to understand👌

  • @AL-kn9wd
    @AL-kn9wd Год назад +2

    without using of AWS access key it will work too, I think you just had to wait few seconds longer. I liked this project and hope to see more!

    • @tuplespectra
      @tuplespectra  Год назад

      Thank you! I'm glad you find the video valuable.

    • @nitropan
      @nitropan 4 месяца назад

      can you tell us how? I also created the role but it does not seem to allot the ObjectPut access.. even with S3FullAccess

  • @mamadoulo1737
    @mamadoulo1737 2 месяца назад

    Great work ! Thank for this project. I have already completed and i learn a lot.

  • @onuhmichael4312
    @onuhmichael4312 9 месяцев назад +1

    Helped me to upskill my knowledge, Awesome tutorial keep it up!!!

    • @tuplespectra
      @tuplespectra  9 месяцев назад

      Glad it helped!

    • @todayspecial9705
      @todayspecial9705 7 месяцев назад

      so how do we make an instance. do we have to pay for this?

  • @ajtam05
    @ajtam05 7 месяцев назад +1

    When choosing the OS for the instance...does it matter what the local machine is? I'm having error issues with trying to call Airflow "airflow standalone" command. There are multiple errors so have to look at each one, but I wanted to see first if selecting the same OS "ubuntu" and other criteria might be the issue?

    • @ajtam05
      @ajtam05 7 месяцев назад

      Issue was the storage size. “ec2 triggerer_job_runner.py:576} info - triggerer's async thread was blocked.....". Thought it was a different issue because there were many error msgs (like Kubernetes not installed). But after googling the issue...finally found out it was storage size of the VM. That resolved it.

  • @andresparra5820
    @andresparra5820 2 месяца назад

    Thank you so much for this tutorial, was very helpful for an interview !!!

  • @jrohit9664
    @jrohit9664 4 месяца назад +1

    The scheduler does not appear to be running. Last heartbeat was received 32 minutes ago.
    The DAGs list may not update, and new tasks will not be scheduled.I have followed all commands you mentioned but i was getting like this in my airflow user interface and the command prompt which i connected to ec2 was running airflow standalone where to run new commands so that i can run airflow scheduler ? pls reply

  • @mohamedelbrawy1222
    @mohamedelbrawy1222 9 месяцев назад +1

    I am very grateful to see this kind of teaching, and I have searched a lot to see a way to connect airflow to the free instance I found a blog on medium that demonstrates that: ( How to Install Apache Airflow on AWS EC2 Instance?)

  • @purnapachhai3021
    @purnapachhai3021 21 день назад

    Very clear explanation

  • @pspointssara8472
    @pspointssara8472 9 месяцев назад

    Excellent presentation. Even though I'm an experienced person still I need to learn a lot from your videos. This reminds me to watch more of your other videos in future. Good work and keep it up.

    • @tuplespectra
      @tuplespectra  9 месяцев назад +1

      Glad it was helpful! Thanks for your comment.

  • @MrBenStringer
    @MrBenStringer 7 месяцев назад +1

    Been following along with this but used GCP instead as it's got $300 free credits for sign up. Might be worth doing that in future?
    Thanks for the great content. Teaching style is brilliant. I'll definitely be checking out your other videos. All the best man.

  • @On-2kd
    @On-2kd 9 месяцев назад +1

    Thanks for this wonderful tuto. It’s time for me to practice now 🙏🏽

    • @tuplespectra
      @tuplespectra  9 месяцев назад

      You’re welcome 😊 . Yes! Practice!

  • @ahm_mask5161
    @ahm_mask5161 Год назад

    Your channel is pure gold

    • @tuplespectra
      @tuplespectra  Год назад

      Thanks so much. I'm glad you find our videos valuable. We hope to continue to provide you with more awesome instructional videos.

  • @ChandanaandSanthosh
    @ChandanaandSanthosh 10 месяцев назад

    Being an aspiring data engineer, this project is really helpful . Thank you so much for this content :)

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      You're very welcome! Thanks for your comment.

    • @sumitrawall
      @sumitrawall 9 месяцев назад

      can it run in free tier EC2 instance?

  • @Mr_EO5
    @Mr_EO5 Месяц назад

    Thanks so much PROF. Your teachings are absolutely amazing 👍 #iJustSubscribed.

  • @sachasmart7139
    @sachasmart7139 Год назад +5

    Really good tutorial. Nicely done. Looking forward to part 2!

  • @narasa12
    @narasa12 5 месяцев назад +1

    Excellent information, thank you so much for posting this video here

  • @collinspo
    @collinspo 2 месяца назад

    This is a masterpiece. Thank you again and again !!!

  • @BI-Rahul
    @BI-Rahul Год назад +1

    This video is an absolute game-changer for anyone looking to build and automate Python ETL pipelines with Airflow on AWS EC2!
    Waiting for part 2!

  • @seth_king_codes
    @seth_king_codes Год назад +2

    amazing man...would love more airflow/dags/python tutorials and also maybe how i can use with scraping data..cheers!

    • @tuplespectra
      @tuplespectra  Год назад +1

      Thanks! We have more videos on airflow you can explore. We will look into your request as well. Thanks!

  • @AlmirTvojDrug
    @AlmirTvojDrug Год назад

    Thank you for this good end-to-end example. However, there are some overheads, e.g. you don't need AWS STS tokens, AWS CLI setup is enough. Also, there is no need to IAM role for EC2. Greetings from Croatia!

  • @RobertoZavala-d1k
    @RobertoZavala-d1k Год назад +2

    congratulation, good job, part 2, dont forget

  • @BI-Rahul
    @BI-Rahul Год назад +8

    I have followed the exact same steps this weekend but this time i am encountering error when I say airflow standalone. The same exact setps I followed last weekend was working flawlessly. Below is the error I am getting this time, Any help would be appreciated. Error:
    pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `dag_id = `. All model fields require a type annotation; if `dag_id` is not meant to be a field, you may be able to resolve this error by annotating it as a `ClassVar` or updating `model_config['ignored_types']`.

    • @totnguyen3308
      @totnguyen3308 Год назад +2

      I also had the same error as you.

    • @raghav01211
      @raghav01211 Год назад +2

      Same for me....Same issue

    • @bukolasalami9021
      @bukolasalami9021 Год назад +1

      I had the same error

    • @bukolasalami9021
      @bukolasalami9021 Год назад +2

      I was able to get pass the error. All I did was to downgrade from pydantic 2.0 to 1.10.10. I hope it works for you

    • @totnguyen3308
      @totnguyen3308 Год назад

      @@bukolasalami9021 How to downgrade from pydantic 2.0 to 1.10.1

  • @Vikasptl07
    @Vikasptl07 6 месяцев назад

    Good beginners video. I am personally not a fan of running your processing code and orchestration on same instance. How do you ensure package dependency,manage virtual environments,resource allocation for different workloads.

  • @ericiyen96
    @ericiyen96 Месяц назад

    Excellent job

  • @Dataengineer-ci8fe
    @Dataengineer-ci8fe 11 месяцев назад

    Thank you so much for the video, just have a question, how can we make the airflow scheduler keep running after exit ssh? I noticed that the airflow scheduler stopped running after a while or if you exit the connection interface to ec2.

  • @Fordalo
    @Fordalo Год назад +24

    my guy is 100% nigerian

  • @chiragmadhukar9504
    @chiragmadhukar9504 9 месяцев назад

    I was enable to open Airflow on EC2 instance, and establish an SSH connection to my Visual Studio. However, my DAG isn't getting created on Airflow. Can you please help?

  • @BambooAlina
    @BambooAlina 7 месяцев назад

    Thank you, this was super helpful!

  • @AbdelmoughitAITYOUB
    @AbdelmoughitAITYOUB Год назад +1

    Thank you for this amazing, well-explained project on apache airflow. I hope you'll make a tuto on apache superset too.

  • @xavierromerocarrion1369
    @xavierromerocarrion1369 3 месяца назад

    Hi bro. Congrats for the video!!
    I have experiencing a major issue, the airflow service and the SSH connection collapses after a few minutes of the instance initiation... What can I do?

  • @Nawar-t56
    @Nawar-t56 3 месяца назад

    Tutorial is really great. However, I am having some issues with loading the airflow UI. When I use ''airflow standalone'', and go to the 8080 port on the url from s3 instance it takes a long time to load the UI. Most times it doesn't load at all. I can't seem to understand what the reason is. Could you please help me?

  • @JuanCruz-nu4mg
    @JuanCruz-nu4mg Год назад +3

    I was able to do this just fine, I had 2 issues though, the t2.small didnt have enough memory, it kept freezing up, so I had to use a t2.medium instance, then it worked perfectly!

    • @tuplespectra
      @tuplespectra  Год назад

      I'm glad all went all for you. Thanks for watching the video.

    • @swarupsaha9451
      @swarupsaha9451 Год назад

      is medium availabe in free tier?

    • @JuanCruz-nu4mg
      @JuanCruz-nu4mg Год назад

      @@swarupsaha9451 unfortunately no

    • @tuplespectra
      @tuplespectra  Год назад +1

      @@swarupsaha9451 No, you will have to pay for using medium, however it is not expensive in as much you did not leave it running extensively.

    • @ritvikrajsingh6228
      @ritvikrajsingh6228 11 месяцев назад +2

      t2.small has low ram thus wasn't able to run airflow ,one alternate solution is to use swap space.

  • @ahmedayodele3299
    @ahmedayodele3299 Год назад

    This tutorial is great. Looking up to more videos. You got a follower.

    • @tuplespectra
      @tuplespectra  Год назад +1

      I'm glad you love it! Thank you! And thanks for subscribing. You can also explore our playlist of Postgresql video series.

  • @Vilayat_Khan
    @Vilayat_Khan 5 месяцев назад

    yay, i managed to finish it! and i have the csv file in s3. thx, u deserve the like lol.

    • @tuplespectra
      @tuplespectra  5 месяцев назад

      Nice work! You did it! Keep learning! Keep growing!

  • @cascadeanalog320
    @cascadeanalog320 2 месяца назад +1

    Thank you very very very Much !

  • @Bindu-h7v
    @Bindu-h7v 9 месяцев назад

    Simply… your the best

    • @tuplespectra
      @tuplespectra  9 месяцев назад

      Thanks so much for your comment. It made my day.

  • @MalvinSiew
    @MalvinSiew Месяц назад

    Hello thank you for the great video. I am running into an issue where the EC2 instance keeps freezing and disconnecting every time I manually trigger a dag. I believe it's because the t2.small instance does not have enough memory but I see this is not an issue for you or others. Just wondering if anyone else is going through the same thing or whether it is something I did wrong? Any help is appreciated, thank you.

  • @bobbytito6301
    @bobbytito6301 Месяц назад

    wonderful video

  • @flaskwater44
    @flaskwater44 11 месяцев назад

    Great tutorial! Thank You.

    • @tuplespectra
      @tuplespectra  11 месяцев назад

      Glad it was helpful! Thanks for your comment.

  • @SonuKumar-fn1gn
    @SonuKumar-fn1gn Год назад +1

    Thank you so much for the video. ❤😊

  • @nikitanikumbh9162
    @nikitanikumbh9162 25 дней назад

    I am able to install airflow on free-tier EC2 instance but not able to connect to : Public IPv4 DNS . can anyone suggest on this ?

  • @emmanuelharel
    @emmanuelharel Месяц назад

    Just a remark: you forgot to tell that here you are using the default vpc. I don't think it is a best practise. I understand the video would be longer but it would worth mentioning it.
    As well depending on how you created your VPC, AWS now can create for you automatically a connection between for ec2 instance and S3.

  • @DEEKSHITH-bw6bo
    @DEEKSHITH-bw6bo Год назад

    I am getting the error at the last step, i.e to store the csv file in the s3 bucket. can you please help me with that
    the log file shows like-- ERROR - Failed to execute job 42 for task transform_load_weather_data ([Errno 22] The provided token is malformed or otherwise invalid.; 54783)

  • @nehalverma1444
    @nehalverma1444 Год назад +1

    This is awesome! Thanks a lot

  • @rohan_mehra
    @rohan_mehra 3 месяца назад +1

    It's like T'Challa is teaching AirFlow! 🤣

  • @guidysoll
    @guidysoll Год назад

    Thank you very much for the amazing content. Already subscribed to your channel.

    • @tuplespectra
      @tuplespectra  Год назад

      You are welcome. Awesome! Thank you for the sub!

  • @jerbear97
    @jerbear97 Год назад

    excellent vibe, i love it. other tutorial make me sleep

    • @tuplespectra
      @tuplespectra  Год назад

      Thanks for your comment. It really means a lot to me.

  • @abhijeetsuryawanshi4064
    @abhijeetsuryawanshi4064 10 месяцев назад +1

    Great video 👍👍👍👍👍

  • @stenardt
    @stenardt Год назад

    Excellent tutorial

  • @yashmatha6180
    @yashmatha6180 8 месяцев назад +1

    I am getting Permission Denied: No AccessKey Presented when i try to run my DAG.
    I have created access keys from my AWS, and have also pasted them in aws_credentials variable. Please Help!!

    • @JayRavalani
      @JayRavalani 8 месяцев назад

      did you find a workaround for this error? i am also facing the same issue.

  • @Dlearn-hr4cb
    @Dlearn-hr4cb Месяц назад

    Since you are using session token, isnt it a manual effort and hence DAG will not be automated?

    • @Dlearn-hr4cb
      @Dlearn-hr4cb Месяц назад

      Since session token will expire right and need to create new one every now and then

  • @shwetabhat9981
    @shwetabhat9981 Год назад

    Amazing !! Looking forward to many more

    • @tuplespectra
      @tuplespectra  Год назад

      I have just released a second part to this where I showed tasks running in parallel. See link here: ruclips.net/video/DKsf88oCPWA/видео.html

  • @gyungyoonpark
    @gyungyoonpark 8 месяцев назад

    I want to make my airflow run every hour, but it seems my token is expired (the one that I get with aws sts get-session-token). how can i get aws credentials that has no expiration?
    for example, each time i run, how can i run aws sts get-session-token and make the result into python variable?

  • @ezechukwufidelis5978
    @ezechukwufidelis5978 Год назад

    you're good. I am looking forward to the next video.

    • @tuplespectra
      @tuplespectra  Год назад

      Thank you so much. I'm glad you find our videos valuable.

  • @AladinAiWisdom
    @AladinAiWisdom 6 месяцев назад +1

    Thanks for the tuto, just as remarque no need for hard coding credentials, since you gave iam role full access and assumed by the EC2. you can write directly in S3 from EC2

    • @tuplespectra
      @tuplespectra  6 месяцев назад

      You're welcome! Thanks for the remark.

  • @jyothikasiri4008
    @jyothikasiri4008 9 месяцев назад

    When import DAG from airflow it shows Module airflow not found can you please guide?

  • @TejaMadire
    @TejaMadire Год назад

    Awesome explanation

  • @ritesh_ojha
    @ritesh_ojha 7 месяцев назад

    @tuplespectra When i try to login airflow by using instance public ip and port 8080. Unable to connect. I have try all things related to security and all

  • @ndthdproduction2900
    @ndthdproduction2900 9 месяцев назад +1

    Hello bro can you explain why airflow throw DAG import error: Broken DAG: [/home/ubuntu/airflow/dags/weather_dag.py] Traceback (most recent call last):
    File "", line 241, in _call_with_frames_removed
    File "/home/ubuntu/airflow/dags/weather_dag.py", line 8, in
    import pandas as pd
    ModuleNotFoundError: No module named 'pandas'
    I installed pandas with sudo pip install pandas but it still there. Could you explain this. Thank you so much for your work !

    • @bizzie14
      @bizzie14 3 месяца назад

      You have to install pandas within the airflow_venv for it to work

  • @kingnguyen6109
    @kingnguyen6109 Месяц назад

    What if I use CSV file instead of API so how can I load the file into airflow file

  • @peterkatongole5984
    @peterkatongole5984 Год назад

    Awesome project. Well explained

    • @tuplespectra
      @tuplespectra  Год назад

      Thank you!

    • @peterkatongole5984
      @peterkatongole5984 Год назад

      If all content creators use this method, showcasing their skills by creating projects, they would inspire so many. Thank you tuplespectra once again.

    • @tuplespectra
      @tuplespectra  Год назад

      @@peterkatongole5984 Thanks for the comment.

  • @khaliltrabelsi5831
    @khaliltrabelsi5831 8 месяцев назад

    i had an error after lunching airflow standalone can you help me please ?
    SqlAlchemySessionInterface.__init__() missing 6 required positional arguments: 'sequence', 'schema', 'bind_key', 'use_signer', 'permanent', and 'sid_length'
    i tried pip install Flask-Session but nothing , i had the same error in my wsl and i solved it the flask-session

  • @manaschauhan2418
    @manaschauhan2418 Год назад

    Amazing explanation, I've tried with Azure VM, if anyone is trying with that installation steps for airflow will be little bit different

  • @ahmeddadjio2003
    @ahmeddadjio2003 10 месяцев назад +1

    great video!!!please can i help me with the installation of airflow because i have a few problems!

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      Thanks! What's the problem you got?

  • @pavanparvathanenii4471
    @pavanparvathanenii4471 5 месяцев назад

    Amazing video❤

  • @ThanmayiR
    @ThanmayiR Год назад

    Access key for openweather is not working when integrated with airflow, but it is working fine if used without airflow. Any reason why this could be happening?

    • @tuplespectra
      @tuplespectra  Год назад

      I'm guessing you are probably making some mistakes. Could you watch the video again and follow it step-by-step? That way you might be able to see what the issue is.

  • @กระเพราะเต้าหู้พิเศษไข่ดาว

    Thank you so much, my father

  • @SanjeevKumar-dr6qj
    @SanjeevKumar-dr6qj Год назад

    thank you for this tutorial

  • @sivaji47
    @sivaji47 6 месяцев назад +1

    Thank you so much

  • @joshuaroberts3987
    @joshuaroberts3987 11 месяцев назад

    when i connected my vscode to my EC2 instance i didnt get all those files in my vscode. just the .py file. i didnt get airflow file or any other files? please help!

  • @RaghulS-nl6wx
    @RaghulS-nl6wx 4 месяца назад

    i get my load_data task failed i configured everything right but still get failed for the last task i couldnt figure it out anyone with the same scenario got any soln?

  • @seth_king_codes
    @seth_king_codes Год назад

    at 58:41 you say to save the python file and it should sync to airflow, ive waited several minutes but dont see any the dag appearing
    how do i ensure hte file is saved? or can i see even where it failed to be uploaded to airflow?

    • @tuplespectra
      @tuplespectra  Год назад

      I suggest you look at you code to ensure there are no mistakes or typo after which restart your airflow server. Let me know if this fix your issue. Thanks for reaching out.

    • @seth_king_codes
      @seth_king_codes Год назад

      ok i didnt restart my airflow server, sounds like thats where i may have gone wrong@@tuplespectra

  • @zahraelhaddi6980
    @zahraelhaddi6980 10 месяцев назад

    thank you so much. i did this on azure but the connection part is different than aws and it didnt work

  • @Mehtre108
    @Mehtre108 8 месяцев назад

    Hello sir,
    Any end to end AWS Data Engineer projects. So that we will tell in interview as 2 yoe.

    • @tuplespectra
      @tuplespectra  8 месяцев назад

      You can work on my projects, understand them and discuss them during interviews.

  • @vaibhavverma1340
    @vaibhavverma1340 Год назад

    It was excellent sessions no doubt, can you please tell us if we want to take multiple country's like, India, America, France, ... etc then how to do that in that case ??????

    • @tuplespectra
      @tuplespectra  Год назад +1

      Check out this video where I showed how to extract for multiple cities ruclips.net/video/ocFzNmgYW9o/видео.html

    • @vaibhavverma1340
      @vaibhavverma1340 Год назад

      @@tuplespectra Thank you so much noticing out my concern :)

  • @RamaDivyaBolla-q9m
    @RamaDivyaBolla-q9m Год назад

    Hi,
    My DAG breaks because of "import pandas as pd". It's giving me below error is airflow:
    Broken DAG: [/home/ubuntu/airflow/dags/weather_api_etl_asp.py] Traceback (most recent call last):
    File "", line 241, in _call_with_frames_removed
    File "/home/ubuntu/airflow/dags/weather_api_etl_asp.py", line 7, in
    import pandas as pd
    ModuleNotFoundError: No module named 'pandas'
    I've pandas installed on my EC2 instance.

    • @tuplespectra
      @tuplespectra  Год назад

      Try sudo pip install pandas if you haven't tried it.

    • @RamaDivyaBolla-q9m
      @RamaDivyaBolla-q9m Год назад

      @@tuplespectra I tried that too, tried uninstalling and installed again, specifically version compatible with python version. Lately I tried some python coding in jupyter lab/notebook and found same import error. Anyone else faced this weird import error? Not just in this project, but while running any py files? Please shed some light on this

    • @bizzie14
      @bizzie14 3 месяца назад

      You have to install pandas within the airflow_venv for it to work

  • @devangpatel1341
    @devangpatel1341 Год назад

    You are clear!!

  • @CodeITData
    @CodeITData Год назад

    MySQL vs PostgreSQL . Which one should i go for as a begineer

    • @tuplespectra
      @tuplespectra  Год назад

      Actually, you can go for either one as a beginner. But I will say since you are already getting the foundations to SQL from my channel using postgresql, I will say to go with postgresql to master the sql skills.

  • @biloloonguesamuel1940
    @biloloonguesamuel1940 Год назад

    hi i can install panda package in airflow container for docker?

  • @mihirgharat7585
    @mihirgharat7585 6 месяцев назад

    my ec2 instance is not loading using a t2 micro instance. could that be the only reason?

    • @tuplespectra
      @tuplespectra  6 месяцев назад

      That's one possible reason. It has happened to me before.

    • @mihirgharat7585
      @mihirgharat7585 5 месяцев назад

      It worked now. But now I’m facing a new issue. My airflow dag is not reflecting on airflow

  • @mudasurbasha5742
    @mudasurbasha5742 10 месяцев назад

    I'm struck at airflow standalone command after running this command it continuessly kept printing like conversation between webserver and triggered. Please help me out

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      I believe it is working fine. You just need to go grab the user name and pw that airflow created for you and enter them on the airflow UI to sign in.

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      Let me know what you get.

    • @mudasurbasha5742
      @mudasurbasha5742 10 месяцев назад

      @@tuplespectra if you could give me your Gmail I can send you some screenshots and I can explain you clearly within the mail where I am getting problem

    • @mudasurbasha5742
      @mudasurbasha5742 10 месяцев назад

      @@tuplespectra to enter username and pw I need to start the airflow right..... By giving the command airflow standalone..... After giving airflow stand alone command the airflow ui is not coming

    • @tuplespectra
      @tuplespectra  10 месяцев назад

      @@mudasurbasha5742 tuplespectra@gmail.com

  • @Dlearn-hr4cb
    @Dlearn-hr4cb Месяц назад

    Where does the airflow folder in vscode came from?

    • @ABIRCHANDA
      @ABIRCHANDA Месяц назад

      it will be created when you install airflow

  • @josecardons6221
    @josecardons6221 4 месяца назад

    amazing bro thanhs

  • @kristiandaclan9236
    @kristiandaclan9236 5 месяцев назад

    If you have problems with installing dependencies it is because instead of sudo apt install3.10-venv, replate it to sudo apt install3-venv to get the latest version. Currently, it's at 3.12

    • @JayPatel-yf7ot
      @JayPatel-yf7ot 4 месяца назад

      Invalid operation upon using that.

    • @buchke_bhai
      @buchke_bhai 4 месяца назад

      This worked thanks

  • @rashedulkabir7479
    @rashedulkabir7479 Месяц назад

    when I ssh using vs code it keeps disconnecting. anyone found a solution?

  • @JayPatel-yf7ot
    @JayPatel-yf7ot 4 месяца назад

    I am getting following error while creating a virtual environment.
    E: Unable to locate package python3.11.9-venv
    E: Couldn't find any package by glob 'python3.11.9-venv'
    If anyone can help then it would be great!!

    • @buchke_bhai
      @buchke_bhai 4 месяца назад

      Hey jay try this sudo apt install3-venv

  • @salmanshikalgar4482
    @salmanshikalgar4482 2 месяца назад

    Command not working sudo pip install pandas in airflow _venv , what to do?

    • @NaijaForexGoddess
      @NaijaForexGoddess 2 месяца назад +1

      Try sudo airflow_venv/bin/python3 -m pip install pandas. It works well for me