Airflow tutorial | Install Airflow | Write and run your first DAG | Apache airflow on Windows Docker

Поделиться
HTML-код
  • Опубликовано: 5 сен 2024
  • How to Install Airflow on Windows
    Airflow local setup
    Create first airflow dag
    Installing Apache Airflow on Windows A Step-by-Step Guide
    install apache airflow on windows with docker
    install airflow
    airflow scheduler
    airflow windows
    airflow on windows
    How to run Airflow on Windows
    airflow interview questions
    airflow interview questions and answers
    apache airflow windows
    install airflow windows
    airflow installation on windows
    install apache airflow on windows
    airflow install windows
    airflow docker windows
    airflow for windows
    apache airflow installation on windows
    install airflow in windows
    install airflow locally windows
    How to set up Airflow on Windows
    install airflow on laptop
    install airflow on pc 2023
    install airflow on computer
    install airflow on windows 11
    install airflow on windows 10
    install airflow on windows with docker
    install airflow
    airflow docker
    docker windows
    apache airflow
    airflow in windows
    airflow windows installation
    install airflow docker windows
    windows airflow
    pip install airflow
    apache airflow github
    install airflow on windows without docker
    airflow on windows 10
    run airflow on windows with docker
    install airflow docker
    How to develop airflow dags on windows
    Airflow hello world dag
    install airflow on windows without wsl
    install airflow on windows without ubuntu
    install airflow on windows server 2016
    can we install airflow on windows
    can i install airflow on windows
    pip install airflow on windows
    How to install Airflow in Windows with Docker
    How to install Airflow using pip
    How can I install Airflow
    install airflow python
    install airflow locally
    install airflow using docker
    pip install airflow providers
    pip install airflow postgres
    How to Install Apache Airflow on Windows without Docker
    Installation of Airflow Airflow Documentation
    Install Apache Airflow on windows 10 without docker
    How to Install Apache airflow in windows machine using
    Apache Airflow Installation on Windows 10
    How to Install Apache Airflow
    Download Docker Desktop for Windows:
    docs.docker.co...
    Download Visual Studio Code:
    code.visualstu...
    Any issues during wsl --update, then please refer below video:
    • How to install Windows...
    dockerfile, docker-compose.yml and welcome_dag.py codes:
    sleek-data.blo...

Комментарии • 124

  • @jonathan.matheus
    @jonathan.matheus Месяц назад

    thanks dude, I spent 2 days trying to find how to do it step by step without any complicated context going on, appreciated

    • @SleekData
      @SleekData  Месяц назад

      Dear Friend, Glad you like them!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, it would be so grateful!
      Thank you for your encouragement.

  • @shafiulamin4128
    @shafiulamin4128 3 месяца назад +1

    Cannot thank you enough. Everything working perfectly. I suppose my Airflow journey has just started .

    • @SleekData
      @SleekData  3 месяца назад

      Dear Friend, Glad you like them!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, it would be so grateful!
      Thank you for your encouragement.

  • @szymonolejniczak7342
    @szymonolejniczak7342 10 месяцев назад +2

    I got here from some course on udemy as i wasnt able to set-up docker with their videos. Your alternate way worked perfectly! thank you very much!

    • @SleekData
      @SleekData  10 месяцев назад

      Dear Friend, Glad it helped!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @NandaAcademies
    @NandaAcademies 5 месяцев назад

    Fantastic tutorial! No one can explain better than this! Of course, I am yet to install docker and airflow, but anyway it should work.

    • @SleekData
      @SleekData  4 месяца назад

      Dear Friend -
      Thank you for your kind words! We're delighted to hear that you're enjoying our content. To stay updated with our future videos, consider subscribing and sharing with your friends. Your support means a lot to us. Thanks again for your encouragement!

  • @mygmail-ti4js
    @mygmail-ti4js 11 месяцев назад +1

    This is really a great Guideline. Thank you for sharing.

    • @SleekData
      @SleekData  11 месяцев назад

      Dear Friend - Thank you your kind feedback! To stay updated with our future content, I kindly request you to consider subscribing and liking our videos. Your support is valuable to us. If you ever need technical support, please feel free to leave a comment on any of our videos, and we'll be sure to respond promptly or create a video addressing the issue.
      If you could share our video/channel with your friends, we would be so grateful!

  • @trunksssz
    @trunksssz 4 месяца назад

    after many tutorial i have watched, only this works on me:) thank youu

    • @SleekData
      @SleekData  4 месяца назад

      Dear Friend, Glad you like them!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @sfcsarah
    @sfcsarah 11 месяцев назад +1

    really looking forward to your video on setting up Postgres

    • @SleekData
      @SleekData  11 месяцев назад +1

      Dear Friend -
      Thank you for your interest! I understand that you're eager to get started, so I'd like to provide you with a high-level overview of the steps required to set up PostgreSQL as your metadata database, even before we create the video tutorial:
      sudo apt install libpq-dev
      sudo apt-get install postgresql
      sudo service postgresql start
      sudo -u postgres psql
      CREATE USER airflow PASSWORD 'airflow';
      CREATE DATABASE airflow;
      GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO airflow;
      Exit
      cd to airflow home
      vi airflow.cfg
      executor = LocalExecutor
      sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@localhost/airflow
      We appreciate your support and would love it if you could like and subscribe to our channel to stay updated with our future content.
      If you could share our video/channel with your friends, we would be so grateful!
      Once again, thank you for your encouragement.

    • @sfcsarah
      @sfcsarah 11 месяцев назад

      @@SleekData Thank you so much! I'll try it out :)

    • @valentinosimeoni6993
      @valentinosimeoni6993 2 месяца назад

      @@SleekData great tutorial! the only one thats worked for me . now im trying to connect postgre with your instructions but isnt working. do i have to change something on the docker-compose.yaml file ?

  • @channuangadi7504
    @channuangadi7504 10 месяцев назад +1

    to the point explanation thank you so much

    • @SleekData
      @SleekData  10 месяцев назад

      Dear Friend - Thank you your kind feedback!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Once again thank you for your encouragement.

  • @user-qy6gk5fs8k
    @user-qy6gk5fs8k 2 месяца назад

    Take a close look at the airflow while watching the tutorial video. Thank you, in progress. I cannot log in with the user name, password, and information in the log in the visul studio code.I previously installed airflow for testing, and can only log in with airflow/airflow information.

    • @SleekData
      @SleekData  2 месяца назад

      Dude - Pls check the video i've shown a file from where you can copy password.

  • @vinodtengli6882
    @vinodtengli6882 9 месяцев назад

    Thank you so much for sharing very useful video

    • @SleekData
      @SleekData  9 месяцев назад

      Dear Friend - My pleasure 😊
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @shafiulamin4128
    @shafiulamin4128 3 месяца назад

    Nice one mate. Very useful

    • @SleekData
      @SleekData  3 месяца назад

      Dear Friend, Glad you like them!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @kumaran3534
    @kumaran3534 Месяц назад

    Helpful!😊

    • @SleekData
      @SleekData  Месяц назад

      Dear Friend, Glad you like them!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, it would be so grateful!
      Thank you for your encouragement.

  • @pushankarmakar1783
    @pushankarmakar1783 10 месяцев назад +2

    theres a problem. at 5:44 in my computer its showing
    ✘ sleek-airflow Error 3.9s
    Error response from daemon: pull access denied for sleek-airflow, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
    The terminal process "C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -Command docker compose -f "docker-compose.yml" up -d --build" terminated with exit code: 1.
    Can you please help me with this
    Edited: nevermind. it was a silly error. im keeping this in case someone makes the same mistake as me. i made the image with the name 'airflowstuff' but used the code given in the description which had the name as 'sleek-airflow' hence it wasnt able o find the image

    • @SleekData
      @SleekData  10 месяцев назад +2

      Dear Friend -
      Did you complete the build step? during that step you are supposed to give a name for the image, I've given sleek-airflow as the name.
      So i have used sleek-airflow:latest in docker compose, what name did you give?
      once after sucessfull build it should locally create sleek-airflow:latest for you...if this is not available locally then it tries in the remote repo and gives you this error.
      Please watch the video once, you will get it.
      Good Luck, please like and subscribe.

    • @franklynchiemekam1682
      @franklynchiemekam1682 8 месяцев назад +1

      Thank you, made the same mistake as well and your correction saved me a lot ❤

  • @azizkanch5257
    @azizkanch5257 4 месяца назад

    Firstly, thanks for the wonderful content. I have followed the steps precisely. However, while trying to do compose up, I am getting this error
    failed to create network airflow-workspace_default: Error response from daemon: failed to update bridge store for object type *bridge.networkConfiguration: read-only file system.
    Please help me. Thank you!

    • @SleekData
      @SleekData  4 месяца назад

      It indicates Docker is encountering a permission issue while trying to create a network for your Airflow workspace.

  • @ThànhNguyễnViệt-f8u
    @ThànhNguyễnViệt-f8u Месяц назад

    Hi Sleek, Im trying to open the localhost but it doesn't response. When View Details, it says "SyntaxWarning: invalid escape sequence '\d'"

    • @SleekData
      @SleekData  Месяц назад

      Dear Friend
      Please double-check the Docker Compose file. You should expose port 8080 from a container to the host. You can refer to the previous video in the playlist.
      Try all possibilities:
      127.0.0.1:8080
      127.0.0.1:8080
      127.0.0.1:8080
      localhost:8080
      localhost:8080
      Also, please open the command prompt and type ipconfig and hit enter. You will get your IPv4 Address. For example, if it is 192.168.56.1, then please try all possibilities with that IP address:
      192.168.56.1:8080
      192.168.56.1:8080
      192.168.56.1:8080

  • @sureshm8687
    @sureshm8687 4 месяца назад +2

    Hi I am facing an issue with localhost 8080 is not responding

    • @SleekData
      @SleekData  4 месяца назад

      Dude - pls try http vs https as well as localhost vs 127.0.0.1
      Also make sure the port 80 / 8080 is forwarded in your docker config files as i did.
      Good Luck, appreciate if you could like and sub.

    • @Templestarz
      @Templestarz 21 день назад

      @@SleekData I have followed the same path as you have shared in your video. The composer file has the port "8080:8080" still it is not working. can you advise what we need to check more

  • @jeffs4810
    @jeffs4810 7 месяцев назад

    Really enjoying this tutorial, but running into an issue on Windows 11. I can get the container to start if I remove the mounted volume from the YML file, but when the volume is included the container fails after a short time with errors that appear to be related to accessing the mounted folder. I can see that the 'airflow' folder is getting created. The error messages in the container log look something like this:
    FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler/2024-01-11'
    ...
    FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler'
    ...
    PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs'

    • @SleekData
      @SleekData  7 месяцев назад

      Friend - pls double check docker files you should switch to correct user ids as i have done.

  • @anhhung4737
    @anhhung4737 14 дней назад

    sorry but my Docker doesn't show any logs and localhost 8080 is not responding.
    Can you help T.T

  • @a05n0pavithra7
    @a05n0pavithra7 3 месяца назад

    I have the bootstrap servers, using the bootstrap servers I wanted to consume messages from a particular topic in those bootstrap servers and using the in-built operator which is ConsumeFromTopicOperator. And the version of Apache Airflow I downloaded is 2.9.1
    Can you please help me with this?

    • @SleekData
      @SleekData  2 месяца назад

      Dude -
      Pls try something like this:
      from datetime import datetime, timedelta
      from airflow import DAG
      from airflow.providers.apache.kafka.operators.kafka import ConsumeFromTopicOperator
      default_args = {
      'owner': 'airflow',
      'depends_on_past': False,
      'email_on_failure': False,
      'email_on_retry': False,
      'retries': 1,
      'retry_delay': timedelta(minutes=5)
      }
      def process_batch(batch):
      # Process batch of messages here
      print("Processing Kafka batch:", batch)
      # Example: for message in batch: do_something(message)
      with DAG(
      dag_id='kafka_consumer_dag',
      default_args=default_args,
      schedule_interval=None, # No scheduling interval as the task executes indefinitely
      start_date=datetime(2024, 6, 14),
      catchup=False
      ) as dag:
      consume_task = ConsumeFromTopicOperator(
      task_id='consume_from_kafka_topic',
      kafka_topic='your_topic_name',
      bootstrap_servers='your_bootstrap_servers',
      group_id='airflow_group_id',
      apply_function_batch=process_batch, # Function to apply to batches of messages
      commit_cadence='end_of_operator', # Commit offsets at the end of the operator
      max_messages=None, # Read to the end of the topic
      max_batch_size=1000, # Maximum number of messages to read in each batch
      poll_timeout=60 # Timeout for polling messages
      )
      Good luck, to stay updated with our future content, I kindly request you to consider subscribing and liking our videos.

  • @kartikeyapande5878
    @kartikeyapande5878 3 месяца назад +1

    localhost:8080 page isnt opening and says "This page isn’t working localhost didn’t send any data. ERR_EMPTY_RESPONSE"

    • @SleekData
      @SleekData  3 месяца назад

      Dude - In you main PC, open command prompt and issue ipconfig command
      and see if you can locate the ip address of your docker (See if you have anything like - Ethernet adapter vEthernet (WSL (Hyper-V firewall)))
      Now use that IP address instead local host, for example you fond the IP address as 192.168.16.1 then try this...
      192.168.16.1:8080/
      also pls try http and https both.
      Good luck, to stay updated with our future content, I kindly request you to consider subscribing and liking our videos.

    • @kartikeyapande5878
      @kartikeyapande5878 3 месяца назад

      This worked. Thanks a lot. 👍🏻
      And yup, have subscribed you already.

  • @alaaeldinmanse5228
    @alaaeldinmanse5228 9 месяцев назад

    I appreciate what you did , and hope you keep going , but there is something i have , the .yml file have more lines than you have in the video , that what i have got from apache airflow website , and i found there is more than on container have been created , Do you have an Explanation for this ? and i can't set my username and password

    • @SleekData
      @SleekData  9 месяцев назад

      Dear Friend - Setup in this video is intended for development and testing purposes.
      But the yml file may be more suitable for higher environments:
      airflow.apache.org/docs/apache-airflow/2.7.3/docker-compose.yaml
      More over the password generally displayed in the log when you start for the very first time. other wise look at the folder path that i have shown in the video for the location of password file.
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @sridharsriii3125
    @sridharsriii3125 2 месяца назад

    In yml file image container was showing error(404) page not found. Any other image link is there
    ("sleek-airflow Error pull access denied for sleek-airflow, repository does not exist or may request 3.4s
    Error response from daemon: pull access denied for sleek-airflow, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

    • @SleekData
      @SleekData  2 месяца назад

      Brother - you have not followed the steps, you have to first build the image locally (sleek-airflow is local image not from repository) since this is not available in your local its trying to download from repository and giving you this error, pls follow the steps side-by-side and step-by-step.
      Good luck.

  • @aruk1040
    @aruk1040 7 месяцев назад

    Thank you so much 😭💘

    • @SleekData
      @SleekData  7 месяцев назад

      Dear Friend, Glad you like them!
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @testapps-tm6uv
    @testapps-tm6uv 2 месяца назад

    Hi there, when I clicked the 8080:8080 link - it leads me to a blank page instead of Airflow admin. I did every step you did and restart the computer as well.

    • @SleekData
      @SleekData  2 месяца назад

      Dear Friend
      Please double-check the Docker Compose file. You should expose port 8080 from a container to the host. You can refer to the previous video in the playlist.
      Try all possibilities:
      127.0.0.1:8080
      127.0.0.1:8080
      127.0.0.1:8080
      localhost:8080
      localhost:8080
      Also, please open the command prompt and type ipconfig and hit enter. You will get your IPv4 Address. For example, if it is 192.168.56.1, then please try all possibilities with that IP address:
      192.168.56.1:8080
      192.168.56.1:8080
      192.168.56.1:8080
      Good Luck, please like and subscribe.

    • @QuangHuy-is7jo
      @QuangHuy-is7jo 2 месяца назад

      @@SleekData thank you

  • @mathandnatureman
    @mathandnatureman 6 месяцев назад

    Hi ,
    I am getting error while generating the build (4:02) like
    "ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c apt-get update && apt-get -y install git && apt-get clean" did not complete successfully: exit code: 100"
    could you please help me out. Thanks.

    • @SleekData
      @SleekData  5 месяцев назад

      pls try to install git and retry.

  • @bishop9168
    @bishop9168 Месяц назад

    The docker file ran successfully but the docker-composer.yml file gives a 'version' obsolete message, any assistance will be appreciated.

    • @SleekData
      @SleekData  Месяц назад

      dude - please check below in your system
      docker-compose --version
      and accordingly try to Update the Compose File Version:
      version: 'x.y'

  • @madtitan44
    @madtitan44 6 месяцев назад

    Hello , I am getting cannot use relative path to connect to sqllite. Please use absolute path

    • @SleekData
      @SleekData  6 месяцев назад

      Dude - Pls doublecheck file mounts in docker file, pls maintain the folder hierarchies just like mine.

  • @user-df1lk5fz3u
    @user-df1lk5fz3u 9 месяцев назад

    THANK FOR THE COURSE
    I tried everything went well until I waited for the dag to appear on the web interface I saw nothing and then if I restarted the yaml the password no longer worked PLEASE SOME HELP FOR ME

    • @SleekData
      @SleekData  9 месяцев назад

      Dear Friend - pls refer the video at 6:10, I have told 2 ways to get your password.
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @mamadoubabadjanballo1977
    @mamadoubabadjanballo1977 5 месяцев назад

    Hi, please i need your help. I can not lunch airflow. this is the error i am getting (
    sleek-airflow Error 3.3s
    Error response from daemon: pull access denied for sleek-airflow, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
    * The terminal process "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command docker compose -f "docker-compose.yml" up -d --build" terminated with exit code: 1.
    * Terminal will be reused by tasks, press any key to close it.

    • @SleekData
      @SleekData  5 месяцев назад +1

      Brother - Looks like the docker image build step was not done or not successful.
      Due that sleek-airflow is not available locally as a result it tries to find in the public docker remote repository.
      Please retry the video step by step, it should work.
      Good Luck, to stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!

  • @ronin959
    @ronin959 6 месяцев назад

    i have a problem - "(root) Additional property coversion is not allowed" (5:42)

    • @SleekData
      @SleekData  5 месяцев назад +1

      Dude - May be you are running airlow with root, pls check docker and docker compose files...
      first try to exactly follow my files and then once every thing is up and running then do your changes on top.

  • @HoangQuocat-xp5eu
    @HoangQuocat-xp5eu Месяц назад

    I do exactly what you do but when i run the final code this happens "airflow.exceptions.AirflowConfigException: Cannot use relative path: `sqlite:///C:\Users\Admin/airflow/airflow.db` to connect to sqlite. Please use absolute path such as `sqlite:////tmp/airflow.db`".

    • @SleekData
      @SleekData  Месяц назад

      Dude - could you try this in airflow.cfg?
      sql_alchemy_conn = sqlite:////C:/Users/Admin/airflow/airflow.db
      if it did not work, pls erase all and start from scratch, step by step and side by side.

  • @MitchMcCall-bd4ul
    @MitchMcCall-bd4ul 4 месяца назад

    the docker image you have is out of date, i am refused access and i cannot find the docker image file on dockerhub

    • @kian69topgun1
      @kian69topgun1 4 месяца назад

      came across the same error myself.. did you find any other image to run ?

    • @MitchMcCall-bd4ul
      @MitchMcCall-bd4ul 4 месяца назад

      @kian69topgun1 no, my advice is dont run airflkw off of docker and find a tutorial that doesnt force you to use docker because its ultimately unnecessary just use a virtual environment in ubuntu

    • @SleekData
      @SleekData  4 месяца назад

      Dear Friend -
      Did you complete the build step? during that step you are supposed to give a name for the image, I've given sleek-airflow as the name.
      So i have used sleek-airflow:latest in docker compose, what name did you give?
      once after sucessfull build it should locally create sleek-airflow:latest for you...if this is not available locally then it tries in the remote repo and gives you this error.
      Please watch the video once, you will get it.
      Good Luck, please like and subscribe.

  • @hemav9981
    @hemav9981 9 месяцев назад

    while running dockerfile,it is showing error.It's showing "error during connect".Please help me.

    • @SleekData
      @SleekData  9 месяцев назад

      Dear Friend - this is error is not sufficient to provide some suggestions.
      May be please restart the PC and retry.
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @IuryLeal-zj8ec
    @IuryLeal-zj8ec 6 месяцев назад

    It didnt workout when i cliked on the link to open the webairflow interface. what can i do?

    • @SleekData
      @SleekData  6 месяцев назад

      Dude - Pls revisit the video and see if you've opened the port in the docker file.
      also try with http vs https and also local host vs 127.0.0.1.
      Good luck, t
      o stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your comment.

  • @rajjoshi5111
    @rajjoshi5111 7 месяцев назад

    hello buddy...thank you for the video...but i am having problem with the localhost:8080 page....it isnt opening and says "This page isn’t working localhost didn’t send any data. ERR_EMPTY_RESPONSE"

    • @SleekData
      @SleekData  7 месяцев назад

      dude - pls try changing between http https?
      Good Luck, to stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

    • @rajjoshi5111
      @rajjoshi5111 7 месяцев назад

      @SleekData thanks but the issue was resolved...I just refreshed the file on docker and it worked

    • @kartikeyapande5878
      @kartikeyapande5878 3 месяца назад

      @@rajjoshi5111 I'm getting same error. Please help to resolve. @SleekData

    • @rajjoshi5111
      @rajjoshi5111 3 месяца назад

      @@kartikeyapande5878 all I did was refresh the file on docker.... And I don't remember doing anything else

  • @basavarajpn4801
    @basavarajpn4801 7 месяцев назад

    Am getting error when running in visual code for welcome_dag.py saying that "No Module named Airflow",
    Please help me with this

    • @SleekData
      @SleekData  7 месяцев назад

      Dude - You can not run in vs code, you need develop in vs code and the execution has to be done in side docker container using airflow UI.

  • @shawnschreier5978
    @shawnschreier5978 5 месяцев назад

    I had set this up perfectly last week and was able to set up some DAGs. However, now I get an error saying localhost did not send any data. Any guidance or help?

    • @SleekData
      @SleekData  5 месяцев назад

      Dude - Pls try http vs https and localhost vs 127.0.0.1
      Good Luck.

    • @sanikabhosale6513
      @sanikabhosale6513 4 месяца назад

      @@SleekData I am also facing the same error tried all the above solutions by yet stuck please help me out

  • @StsGamin
    @StsGamin 6 месяцев назад

    Hy there is a problem in launching the UI as its not connecting. I tried to delete and rebuild the image, tried https still its not connecting. What to do ??

    • @SleekData
      @SleekData  6 месяцев назад

      Dude - did you try opening the port in docker compose file?
      ports:
      - "8080:8080"
      also pls check if there are any errors in the docker log?

    • @deepaksinghchauhan489
      @deepaksinghchauhan489 6 месяцев назад

      @@SleekData Hi bro I also face the same issue, my 8080 port is in use for some other thing, So I enter
      9000:9000 and then I ran, the container was built and working properly but when I clicked on for UI . It shows "This page Isn't working" and also, the login interface is not coming in my browser for airflow login.
      Although password is showing in my airflow folder.
      please help me in resolving this issue.

    • @ananddarji86
      @ananddarji86 3 месяца назад

      Same issue i am facing @sleekdata

  • @SouhaCherif-vp2vw
    @SouhaCherif-vp2vw 5 месяцев назад

    hey i want to use sudo command and it keeps asking me to write the airflow password, but i cant seem to find it anywhere

    • @SleekData
      @SleekData  4 месяца назад

      Dude - Please play the video side by side may be on another device/phone and check, it should work.
      Good luck.

  • @Wolololo_Wololo
    @Wolololo_Wololo 6 месяцев назад

    Hi, When I compose up, it shows the DAG Import Errors at the UI
    I used some module in the DAG and I installed the module locally. How to install the module into docker container?
    Broken DAG: [/opt/airflow/dags/podcast-summary.py] Traceback (most recent call last):
    File "", line 219, in _call_with_frames_removed
    File "/opt/airflow/dags/podcast-summary.py", line 7, in
    import xmltodict
    ModuleNotFoundError: No module named 'xmltodict'

    • @SleekData
      @SleekData  6 месяцев назад +1

      Hi Dude - pls watch the next video, i have also given the link below, it talks about similar error and 2 ways to address it.
      ruclips.net/video/OuRiX1XQgyY/видео.html
      Good luck, pls like and subscribe to stay up to date.

  • @abinashkumar1533
    @abinashkumar1533 6 месяцев назад

    these three tasks has been executed successfully , But where these tasks is excuted I do not know . How I executed this task I do not know. Just I know I have executed these tasks.
    A little knowlege is dangerous thing.

    • @SleekData
      @SleekData  6 месяцев назад

      Hi Dude - This is playlist, please watch full playlist you would get better understanding, it is just around 1 hour.
      ruclips.net/p/PLc2EZr8W2QIAI0cS1nZGNxoLzppb7XbqM&si=iAK3qM2qcZ69LgSi

  • @kxt66
    @kxt66 2 месяца назад

    Thank you very much for the video. I followed the instructions, but I encountered a problem.
    When I commend “Compose Up” for the docker-compose.yml file I get the following message :

    ✔ Network airflow_default Created 0.3s
    - Container airflow-sleek-airflow-1 Starting 0.5s
    Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:8080 -> 0.0.0.0:0: listen tcp 0.0.0.0:8080: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

    This is the first problem.
    The second problem is, when I change port to "8081:8081" in docker-compose.yml, the container is being created, bot when I try to open it in web browser I get the message : The connection was reset.
    I admit, that I am not very familiar with ports configuration issue.
    How can I fix the problems described above ?
    Thanks a lot for the help 😊

    • @SleekData
      @SleekData  2 месяца назад

      Dude -
      Restart and retry if still issue persists.
      try this in cmd and find the PID that is currently using 8080
      netstat -ano | findstr LISTENING
      then go to task manager and process tab, right click on any column heading and enable PID, with this you can identify what is locking your 8080.
      Good luck, to stay updated with our future content, I kindly request you to consider subscribing and liking our videos.

    • @kxt66
      @kxt66 2 месяца назад

      @@SleekData Thank you very much for your help. It works parfectly and I will follow the next videos :) Subscribed of course :)

  • @user-wm1hz8jm6z
    @user-wm1hz8jm6z 9 месяцев назад

    I have a problem in the parte of cmd could you tell me how i can do that

    • @SleekData
      @SleekData  9 месяцев назад

      Dear friend
      I could you understand your problem, pls add a new command elaborating the issue.
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

  • @ahmedtr4305
    @ahmedtr4305 5 месяцев назад

    Could I schedual execute python script located in my local machine?

    • @SleekData
      @SleekData  5 месяцев назад

      Dude - To execute a Python script in Apache Airflow, you typically use the PythonOperator provided by Airflow.
      Please search around that you will get an idea.
      Good luck., to stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your engagement.

  • @A_AMARI
    @A_AMARI 6 месяцев назад

    hi sir, can you please check on this , when i click on the compse up it shows me this error " sleek-airflow Error 2.8s
    Error response from daemon: pull access denied for sleek-airflow, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" can you help me please

    • @SleekData
      @SleekData  6 месяцев назад

      Dude - sleek-airflow is build locally in the previous step, pls check it since its not available locally it is searching in the remote repo.
      Good luck, To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

    • @A_AMARI
      @A_AMARI 6 месяцев назад

      thanks a lot@@SleekData

  • @hurstilthymy4943
    @hurstilthymy4943 5 месяцев назад

    I am getting a
    Import "airflow.operators.python_operator"
    How can I fix this?

    • @YukeshAS
      @YukeshAS 5 месяцев назад

      instead of python_operator use python, hopefully that will fix it!

    • @SleekData
      @SleekData  4 месяца назад

      pls try this - from airflow.operators.python_operator import PythonOperator

  • @abubakarsiddiquem9358
    @abubakarsiddiquem9358 6 месяцев назад

    Are these steps same for macOs?

    • @SleekData
      @SleekData  5 месяцев назад

      Dude - Once you install docker and vs code for mac, the remaining steps are almost same. Good Luck.

  • @muhammadfaridansutariya4411
    @muhammadfaridansutariya4411 9 месяцев назад

    can i implement this tutorial on production sir?

    • @SleekData
      @SleekData  9 месяцев назад +1

      Dear Friend - This setup is intended for development and testing purposes, employing SQLite as the metadata database and the Sequential Executor. However, it is not recommended for production environments.
      For production use, it is advisable to use PostgreSQL metadata database and a more robust executor. Alternatively, in a cloud environment, consider utilizing managed Apache Airflow instances like AWS Managed Workflows for Apache Airflow, Google Cloud Composer, or the Airflow integration within Azure Data Factory.
      Also refer below yaml file to get fair idea about production kind of deployments.
      airflow.apache.org/docs/apache-airflow/2.7.3/docker-compose.yaml
      To stay updated with our future content, I kindly request you to consider subscribing and liking our videos.
      If you could share our video/channel with your friends, we would be so grateful!
      Thank you for your encouragement.

    • @muhammadfaridansutariya4411
      @muhammadfaridansutariya4411 9 месяцев назад

      can u give me reference tutorial deployment airflow on production sir?@@SleekData

    • @SleekData
      @SleekData  9 месяцев назад

      Bro - Are you not keen into managed Apache Airflow instances like AWS Managed Workflows for Apache Airflow, Google Cloud Composer, or the Airflow integration within Azure Data Factory?
      You can bring up scalable Airflow in mater of 5 to 10 minutes with these services.. and they are mostly pay as you go cloud model.
      Please note if you want to implement manually there are many things you need to consider:
      Executors (Workers)
      Metadata database
      Not sure what you have decided for these?
      Note: Please add new comment every time, because i do not get notifications on existing comments, so i some time miss to reply..

  • @sultanchissonobie8376
    @sultanchissonobie8376 11 месяцев назад

    what version of python do you use?

    • @SleekData
      @SleekData  11 месяцев назад

      Dear Friend - Currently the apache/airflow:latest and apache/airflow:2.5.2 images are Python 3.8 images.
      Any time you can reach out to this page to get latest information:
      airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html
      Good Luck, we appreciate your support and would love it if you could like and subscribe to our channel to stay updated with our future content.
      Also if you could share our video/channel with your friends, we would be so grateful!

  • @mohsinrajpoot3780
    @mohsinrajpoot3780 10 месяцев назад

    where is content for dockerfile

    • @SleekData
      @SleekData  10 месяцев назад

      Dear friend - Here you go, you can also find the link in description:
      sleek-data.blogspot.com/2023/09/how-to-install-airflow-on-windows.html
      Good luck, pls don’t forget to hit like and subscribe.

  • @krishnavelamala7853
    @krishnavelamala7853 2 месяца назад

    getting this error while executing the dag:
    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.quotable.io', port=443): Max retries exceeded with url: /random (Caused by NameResolutionError(": Failed to resolve 'api.quotable.io' ([Errno -2] Name or service not known)"))

    • @HoangQuocat-xp5eu
      @HoangQuocat-xp5eu Месяц назад

      same error. Have you fixed it yet?

    • @SleekData
      @SleekData  Месяц назад

      looks strange, may please start from scratch, step by step and side by side.