Apache Airflow One Shot- Building End To End ETL Pipeline Using AirFlow And Astro

Поделиться
HTML-код
  • Опубликовано: 31 дек 2024

Комментарии • 61

  • @krishnaik06
    @krishnaik06  2 месяца назад +3

    Check out my new Udemy complete MLOps Course with 10+ end to end ML Projects:
    www.udemy.com/course/complete-mlops-bootcamp-with-10-end-to-end-ml-projects/?couponCode=SHARING

    • @adityapatel907
      @adityapatel907 2 месяца назад

      sir, we need new udemy coupon for ' DSA bootcamp + Leetcode Exercise ' . please sir ! do it quickly.🙏

  • @tanmaybagul2957
    @tanmaybagul2957 2 месяца назад +5

    Realizing the Power of Airflow 😎 this video help me have high level understanding of what i was doing from last 3 years

  • @SandipJadhav-ls1kg
    @SandipJadhav-ls1kg 2 месяца назад +14

    when will be big data engineer cource we can expect krish sir

    • @gagansingh3481
      @gagansingh3481 2 месяца назад

      It's Etl , etl can be done though power query and load it to postgre sql .

    • @gagansingh3481
      @gagansingh3481 2 месяца назад

      It's Etl , etl can be done though power query and load it to postgre sql .

  • @mallorypiedmont3021
    @mallorypiedmont3021 3 дня назад

    Thank you! This was a very helpful video!!

  • @oceanview3165
    @oceanview3165 Месяц назад

    It's very straightforward and helpful! Thank you!

  • @jaggyjut
    @jaggyjut 2 месяца назад +2

    Awesome. How is airflow different in comparison to Microsoft fabric

  • @bp2807
    @bp2807 2 месяца назад +3

    Krish Sir you really bring very good course, please include kubernetes. Also which tablet and software you use to write and explain sir. Love the way you teach. Kindly answer🙏

  • @madhuppandey3151
    @madhuppandey3151 2 месяца назад +2

    Hi krish,
    How many videos you will upload on this series ?

  • @BechirMathlouthi-n9h
    @BechirMathlouthi-n9h Месяц назад

    very insightful !! thanks for the effort !

  • @mohitjoshi8984
    @mohitjoshi8984 2 месяца назад +4

    Sir talk about Kubernetes please

  • @bralabala
    @bralabala 2 месяца назад +3

    what I would have preferred to see is how you configured it to work with local files rather than calling an external api.
    since I posted this comment, I have now solved the problem myself.
    the short of it is to use astro's docker-compose.overrride.yml file to mount your local drives so it can communicate with the local directories

  • @tajudeensanni9240
    @tajudeensanni9240 2 месяца назад +1

    Krish! This is amazing!!! Please is there any way Apache Airflow can be connected with k8

  • @tusharmodi8344
    @tusharmodi8344 2 месяца назад +4

    Sir for your mlops course do we also need to have aws subscription to do project work or is it not needed we can do something in free tiwr

    • @krishnaik06
      @krishnaik06  2 месяца назад

      U can do in aws free tier

    • @harshsinha12
      @harshsinha12 2 месяца назад

      ​@@krishnaik06sir the AWS Environment set-up is giving some errors, i have tried various fixes

    • @ARkhan-xw8ud
      @ARkhan-xw8ud 2 месяца назад

      @@harshsinha12 i have taken of my friend

  • @dineshbiswal2775
    @dineshbiswal2775 2 месяца назад +1

    What tools are you using for the video?

  • @maxwellmaxwels9618
    @maxwellmaxwels9618 Месяц назад

    Top content, Classic!

  • @aliensamv3997
    @aliensamv3997 2 месяца назад +1

    hey Krish please add production-level project in your paid course (399) ... there were only basic project

  • @ravulapallivenkatagurnadha9605
    @ravulapallivenkatagurnadha9605 2 месяца назад +1

    Please add more Videos on Data Engineering

  • @tejasgandhi7045
    @tejasgandhi7045 2 месяца назад

    Thank you for such a nice video. It was really informative. I want to understand from the multiple pods which are coming up, if I want to run the worker node in AKS then what all chants are required and the process ?

  • @aibutsimple
    @aibutsimple 2 месяца назад +2

    Need more etl tools using project

  • @sachinkumar8369
    @sachinkumar8369 2 месяца назад +1

    hey krish! thank you so much for this helping video. Can we use this build this using aws free tier account?

  • @satyamsubhaprakash
    @satyamsubhaprakash 2 месяца назад +1

    Hi Sir, can you please add projects on Market mix modelling?

  • @THEINDIANSAVIOR
    @THEINDIANSAVIOR 2 месяца назад

    Amazing sir 👏👏

  • @saifilicious1749
    @saifilicious1749 2 месяца назад +1

    Thanks bhai you listen to me

  • @ambivert_sigma_motivation
    @ambivert_sigma_motivation 2 месяца назад

    Thanks for this course! Can you create a Jarvis using Python+AI?

  • @Beliver5894-l8h
    @Beliver5894-l8h 2 месяца назад

    Hello Krish,
    create a end to end data science project which should involve ETL steps as well with airflow and also deployment

  • @aliensamv3997
    @aliensamv3997 2 месяца назад +1

    hey krish please add production level project in your paid course (399) ...

  • @ChandanNayak-lq1pd
    @ChandanNayak-lq1pd 2 месяца назад

    Great ContentSir,will it be the part of data engineering course,and when is it available?

    • @krishnaik06
      @krishnaik06  2 месяца назад +2

      yes a;ready started big data engineering course recording

    • @sumitrukadikar3521
      @sumitrukadikar3521 2 месяца назад

      ​@@krishnaik06sir when it will launch?

  • @baophan259
    @baophan259 17 дней назад

    Can the pipeline still run when i turn off my laptop ?

  • @aliensamv3997
    @aliensamv3997 2 месяца назад

    hey krish please add production level project in your paid course (399) ... how we are suppose to know working process at industry lvl ... just 1 project please

  • @Hahaha-y9x
    @Hahaha-y9x Месяц назад

    Running Dockers and airflow in local windows? It needs Linux environment right?

    • @anushak1446
      @anushak1446 5 дней назад

      no can be done in windows also

  • @istiyakahamad2114
    @istiyakahamad2114 2 месяца назад

    Great work..

  • @hemanthvarma1939
    @hemanthvarma1939 2 месяца назад

    Your Voice Was Breaking in the Video due to a Connectivity issue in your end...Does this happen in any-part of your Udemy Video's too...Planning to Buy the Data Science Course on Udemy

    • @krishnaik06
      @krishnaik06  2 месяца назад

      U can get it all the videos are recorded properly

    • @hemanthvarma1939
      @hemanthvarma1939 2 месяца назад

      @@krishnaik06 Thanks a Lot Krish Sir

  • @MOHIT836
    @MOHIT836 2 месяца назад +4

    Krish sir, is this available on MLOps course ??

    • @krishnaik06
      @krishnaik06  2 месяца назад +2

      yes many projects have been uploaded there :)

  • @Sankarm018
    @Sankarm018 2 месяца назад +1

    Sir pls provide full DATA science course from beginner to advance level on udemy sir

  • @alhasanmohammedbinquraish960
    @alhasanmohammedbinquraish960 2 месяца назад +2

    Hindi video sir airflow 🙏🙏🙏

  • @RakeshPrajapati-b4z
    @RakeshPrajapati-b4z 2 месяца назад

    Sir machin learning ka paid cours kaise join kare kaise sir please guide me

  • @mohammadfurquan241
    @mohammadfurquan241 2 месяца назад

    📌ONE VIDEO ON KAFKA ALSOOOOO

  • @prathmeshatre6251
    @prathmeshatre6251 2 месяца назад

    S missing in using word

  • @siddharthtyagi1254
    @siddharthtyagi1254 2 месяца назад +1

    sir please upload some more tools in udemy course

    • @krishnaik06
      @krishnaik06  2 месяца назад +1

      yes brother kubernetes will also get added

    • @siddharthtyagi1254
      @siddharthtyagi1254 2 месяца назад +1

      @@krishnaik06 sir i want to learn feature store also from you . please sir

  • @njagikevin1069
    @njagikevin1069 2 месяца назад +1

    Env file ".env" found. Loading...
    [+] Running 3/4
    - Container 20241028-etl-weather_43a1dc-postgres-1 Starting 1.2s
    - Container 20241028-etl-weather_43a1dc-triggerer-1 Recreated 0.4s
    - Container 20241028-etl-weather_43a1dc-scheduler-1 Recreated 0.5s
    - Container 20241028-etl-weather_43a1dc-webserver-1 Recreated 0.4s
    Error: error building, (re)creating or starting project containers: Error response from daemon: Ports are not available: exposing port TCP 127.0.0.1:5432 -> 0.0.0.0:0: listen tcp 127.0.0.1:5432: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
    i am getting this error when running astro dev start

    • @blazemates
      @blazemates Месяц назад

      go to services and turn off you postgress service. or see if any other image is using that port

  • @gillfromamritsar
    @gillfromamritsar Месяц назад

    I got the point it's base one but I have some senerios to thinnk about
    -> What is Retry mechanism if any of the step fails
    -> If we have a senerio where a large data should be loaded then from one DB to another then how to resume from a point we left of, for example there are 50k records to be loaded from one loc to another then and process broke at 25k then it should resume when next time it runs instead of running 50k from point 0

  • @AlHasan-sh3il
    @AlHasan-sh3il 9 дней назад

    Sir , please hindi video 😭🙏

  • @KNfarming882
    @KNfarming882 2 месяца назад +1

    First comment

  • @AlHasan-sh3il
    @AlHasan-sh3il Месяц назад

    sir hindi video please

  • @deepop1287
    @deepop1287 2 месяца назад

    Hello sir , Actually i want to buy your Udemy course of DATA ANALYST , But the price is currently high , so can you please provide me a Udemy coupon for this course then it will be really helpful sir 🥺🥺