arrgh .. spent about 3h trying to figure this out, basically all the online-instructions missed one small bit or another ... with your instructions, le voila, it works straight up. Thanks a lot!
Fabulous! Thanks Marc! I have installed Airflow2.0 successfully. The webserver was failed to start in my MAC. But after I have increased the memory to 4GB ...it works.
@@kikecastor Thanks for your response Armonia. I was able to install airflow 2 with wsl but when I create a dag and try to debug in VS I am getting error : ModuleNotFoundError: No module named 'airflow'
how to get the compose file with the latest images? Is the doc updated regularly or I grab latest images manually from docker hub and customize my compose file
When I try to run the official 2.0.1 docker-compose.yaml file at airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml on my Ubuntu 18.04 LTS I get the following error: ERROR: The Compose file './docker-compose.yaml' is invalid because: Invalid top-level property "x-airflow-common". Valid top-level sections for this Compose file are: services, version, networks, volumes, and extensions starting with "x-". You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1. For more on the Compose file format versions, see docs.docker.com/compose/compose-file/ services.airflow-init.depends_on contains an invalid type, it should be an array services.airflow-scheduler.depends_on contains an invalid type, it should be an array services.airflow-webserver.depends_on contains an invalid type, it should be an array services.airflow-worker.depends_on contains an invalid type, it should be an array services.flower.depends_on contains an invalid type, it should be an array Changing the version to 3.4 removes the first error but I still get docker complaining about depends_on. How can I fix it? My docker-compose version is docker-compose version 1.17.1, build unknown docker-py version: 2.5.1 CPython version: 2.7.17 OpenSSL version: OpenSSL 1.1.1 11 Sep 2018 while docker version is Client: Version: 19.03.6 API version: 1.40 Go version: go1.12.17 Git commit: 369ce74a3c Built: Fri Dec 18 12:21:44 2020 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.6 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: 369ce74a3c Built: Thu Dec 10 13:23:49 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.3.3-0ubuntu1~18.04.2 GitCommit: runc: Version: spec: 1.0.1-dev GitCommit: docker-init: Version: 0.18.0 GitCommit: Thanks in advance, Flavio
Great tutorial Old but relevant. Thanks! Marc, I am using Visual Studio Code and everytime I want to save my dag file, I need click a button "Retry as Sudo". Can you tell me what to do here... it is quite annyoing! Regards!
Hi Marc, great tutorial. Airflow is running w/o Problems. I tried to use vs code with airflow and found your new video "Configure VS Code to Develop Airflow DAGs with Docker at ease!" However, I don't understand where the Dockerfile come into the picture. Can you please elaborate! ---> Reopen in Container looks totally different as in your video. Thanks
Question: what's the recommended way to increase the number of celery workers using docker compose? Say from 2 workers to 10 workers? Copy&paste worker keys in docker compose yaml files?
Hi, I'm facing the issue PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/20201-03-16' when running "docker-compose up airflow-init" . Any idea? Thanks
@@DaniloPako Hi, I just solved it without using sudo. You just have to make sure all of the files you create like dags, logs, plugins and the folder you're currently in, the owner and the groupid of them are the user you've log in.
I have many problems using PythonVirtualenvOperator or ExternalPythonOperator inside docker because you must include system site packages as True (it creates conflicts between venv and base python libraries) or otherwise you will get "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv"
Hi Marc, thanks for your sharing! I'd like to know how to install third-party modules. When I installed yfinance module, there was a dag import error : no module named yfinance.
Thanks for the wanderful tutorial. I understood that the DAG file I stored in DAG folder will be added to Airflow. But what happen if Airflow is running in remote docker that I only have web access? I can upload DAG from my local disk to remote? Or is there other way to do it?
Hi I'm a noob I'm using the same YAML file but after running the command "docker-compose up airflow-init" on my ubuntu machine I'm getting this error please help. ERROR: The Compose file './docker-compose.yaml' is invalid because: Invalid top-level property "x-airflow-common". Valid top-level sections for this Compose file are: services, version, networks, volumes, and extensions starting with "x-". You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1. For more on the Compose file format versions, see docs.docker.com/compose/compose-file/ services.airflow-init.depends_on contains an invalid type, it should be an array services.airflow-scheduler.depends_on contains an invalid type, it should be an array services.airflow-webserver.depends_on contains an invalid type, it should be an array services.airflow-worker.depends_on contains an invalid type, it should be an array services.flower.depends_on contains an invalid type, it should be an array
Thank you for the awesome tutorial. I do have one question though: how can I install python packages with docker-compose when creating the containers? for example I would like to install Pymongo.
@@ramsescoraspe yes, but how do I install a python library like PyMongo, or OpenCV in the container? PythonVirtualenvOperator allows for functions/methods to be created including the module imports they need and then they are destroyed, but I do not have those modules installed in the container. Until now, each time I installed Python modules in containers I did it with a help of a Dockerfile (e.g. inside the Dockerfile I enter "RUN pip install opencv-python") but it is not clear to me how to do the same using a docker-compose.yaml file. Edit: figured it out: I had to add a pointer to the Dockerfile in the docker-compose.yaml
3 года назад
@@derzemel hi, how do you do this ? (add a pointer)
@ in my case, the airflow webserver service is build like this (the Dockerfile is in the same dir as the compose): airflow-webserver: build: context: . dockerfile: Dockerfile
@@derzemel do you know how can I add Airflow dependencies inside the docker-compose.yaml file? Also is there a way to provide access to my AWS resources, such as S3, either on the yaml file on the Airflow UI?
anyone know why I can't access the installed airflow docker-compose in ec2 instance via browser? I have installed airflow using docker-compose in ec2 instance, all containers running, I have set inbound rules security group TCP 8080 port to be accessible. But when I open ec2dnsaddress:8080 on the browser, it shows This site can't be reached. I have check it also in docker-compose logs airflow-webserver, it doesn't capture access from the outside and it only logs healthcheck
Thank you Marc. I was in hurry to find out how to run airflow and kept failed somehow. However with your nice clear explanation, nothing is mysterious anymore~
Hello guys. Just a tip for everyone: Do not try to create the airflow folder outside your folder user... You will run into permissions problems (a tried to create a Airflow folder o /opt/, but i strong don't recommend that).
Great video Marc. It's sort of crazy how easy that was (even on WSL2)... Thank you Some things I'm still considering afterwards: 1) Is this enough for a production deployment of Airflow if the database was decoupled from the rest of the container? If the container crashed for whatever reason all of the connections would be lost, so separating is a good idea. 3) For local testing/debugging of an instance I'm going to try and mount DAGS that exist in another project folder instead of the one that we created. 3b) For local testing I might also try and store connection details in environment variables in the .env file rather than relying on the persistence of the database.
I had to increase the amount of RAM available to Docker to 6GB for this to work on my Mac. Also had to enable permissions to the folder i worked in with CHMOD.
Hi, could you please let me know how you enabled permissions using CHMOD? I keep getting the following error: "OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "version": executable file not found in $PATH: unknown"
Hi, thank you for the tutorial, it is very clear and concise, however im facing an issue in which i can't connect to the airflow app through localhost:8080 (browser says empty response). What i've noticed in the logs is that both redis and postgres starts and immediately shuts down and then restart again and so on. For example redis: "redis_1 | 1:signal-handler (1618756176) Received SIGTERM scheduling shutdown..." and for postgres is: "postgres_1 | 2021-04-16 21:57:28.312 UTC [47] LOG: received fast shutdown request" i don't know is someone has faced this issue before, i'm running docker and docker-compose on mac. Any help would be very appreciated
probably you don't need this now - but for anyone who comes across the same issue, it is possible your docker version needs updating to at least v2. If you have v2 already, you may need to allocate more memory (>=5GB to the Docker Engine)
Great tutorial! Short and sweet. I followed the exact same steps and checked the containers status, redis and postgres were healthy but airflow-scheduler, flower, worker, webserver and triggerer were unhealthy then I deleted all the containers and repeated all the steps and now I'm getting error as "database "airflow" does not exist". Redis and postgres containers are running without any problem. I would appreciate if you can help me understand the error. Thanks.
Hi guys, I got this problem same as you. I am operating in Windows 10. Instead of applying "echo -e....." command, I created a .env file on same directory as .yaml fileAIRFLOW_UID=50000 in it. Problem was solved!
I followed the steps mentioned here, but getting no response from gunicorn master within 120 seconds and the webserver keeps getting restarted. Can anyone help with any lead here please?
Very cool! Unfortunately, I got an error saying 'port 5555 is already allocated..' but I am pretty sure there is nothing on there. So, not sure what's going on.
Hi Marc I installed docker desktop at windows using Ubuntu wsl. I changed the dags directory path in .yaml file to my c:\ drive folder in windows. When I start web UI, it doesn't pick my dags.py file. what can be the issue.
For me it does not work. Docker compose is creating path "./local" and I can not access it. Airflow can not read my DAGS. it is very frustrating. I have been installing airflow for 8th time and none of them worked...
While running Airflow 2 via Docker Compose(Just like the above video), I am unable to successfully execute DockerOperator tasks. Can you enlighten with a video reference or doc reference about how to properly configure Airflow Docker compose file or Docker Operator to run tasks
When I got to the docker-compose up airflow-init, I get the following error: "Python-dotenv could not parse statement starting at line 1 Traceback (most recent call last): File "docker\api\client.py", line 214, in _retrieve_server_version File "docker\api\daemon.py", line 181, in version File "docker\utils\decorators.py", line 46, in inner " A few dozen more error lines afterwards, but I can't make it work so far
does not do the magic in Windows. beside the curl command not working, the docker-compose up airflow-init command says: top level object must be a mapping.
When i followed the steps and installing, I am getting the error " manifest file not found". I have seen this error is reported by others as well; I changed the image to 2.0.1 and it worked by later I get an error about old version used.
Hi mark. I bought your course but I got an error trying to run the bash operador that insert data into the user table. I have tried the comand alone in the console and it works but when I used inside my dag in my bash operator I got this error bash command failed. The comand returned a non-zero exit code. I have tried a lot but I still can't found a solution for this
Hello Marc, I have recently installed airflow 2 using docker compose file as suggested in this video. But, when I enhanced the dag with mutiple connections i.e., Gdrive->S3, S3->Snowflake,Snowflake->S3 operations using pyspark and sql scripts, the webserver keeps restarting and at times shows unhealthy. Can you please suggest or advice what could have gone wrong or should I consider increasing docker memory?
The first time I run "docker-compose up airflow-init", everything is okay. But after I run "docker-compose down" and then I run "docker-compose up airflow-init" once again, I get the message "container for service "postgres" is unhealthy" and the airflow-init container fails. I have to run "docker-compose up airflow-init" once more time to start airflow-init container. Does anyone get the same problem like me? Could you give me some advice to avoid this, please? Thanks all!
First of all, thank you so much for this *awesome* video. It is really helpful. I followed this tutorial and was able to access AirFlow seamlessly. But I want to have apache-airflow-providers operators. So, I tried giving them in _PIP_ADDITIONAL_REQUIREMENTS and also building using Dockerfile. But nothing worked and I still see "error: command 'gcc' failed with exit status 1". I changed airflow image to 2.1.2-python3.7 as slim versions don't include extra libraries. But no luck. Could you help me resolve this issue?
Anyone facing problem in getting the logs displayed in the UI? Clicking a task --> Log --> gives me a blank log frame. However, it allows me to download it to my machine.
The only of many tutorial that actually helped! Thanks
arrgh .. spent about 3h trying to figure this out, basically all the online-instructions missed one small bit or another ... with your instructions, le voila, it works straight up. Thanks a lot!
Wow, its amazing how far the Airflow team has come with this. Thanks Marc!
Thanks Mike
Fabulous! Thanks Marc! I have installed Airflow2.0 successfully. The webserver was failed to start in my MAC. But after I have increased the memory to 4GB ...it works.
how did u increase your mac memory
For anyone coming here from the 2024's and beyond, in Linux, specifically Ubuntu, remember to use: `docker compose up init-airflow`
hi, ive tried the using the command however the error still there. mind to help? thank you!
It works on Windows and Mac, I tried both and it works, thanks (on Windows with some tricks)
Please can you share the tricks on windows..I tried on windows its not working for me. Please do reply will be very helpful
@@anjanashetty482 for windows use the wsl tool to run the commands described in the video
@@kikecastor Thanks for your response Armonia. I was able to install airflow 2 with wsl but when I create a dag and try to debug in VS I am getting error : ModuleNotFoundError: No module named 'airflow'
@@anjanashetty482 are you in the correct environment?
@@kikecastor Yes I am, do I have to explicitly do pip install apache-airflow
I struggled to get airflow running for a long time and this short video helped me SO MUCH, thank you!!
Happy to help!
how to get the compose file with the latest images? Is the doc updated regularly or I grab latest images manually from docker hub and customize my compose file
use the doc yes :)
When I try to run the official 2.0.1 docker-compose.yaml file at airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml on my Ubuntu 18.04 LTS I get the following error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
Invalid top-level property "x-airflow-common". Valid top-level sections for this Compose file are: services, version, networks, volumes, and extensions starting with "x-".
You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see docs.docker.com/compose/compose-file/
services.airflow-init.depends_on contains an invalid type, it should be an array
services.airflow-scheduler.depends_on contains an invalid type, it should be an array
services.airflow-webserver.depends_on contains an invalid type, it should be an array
services.airflow-worker.depends_on contains an invalid type, it should be an array
services.flower.depends_on contains an invalid type, it should be an array
Changing the version to 3.4 removes the first error but I still get docker complaining about depends_on. How can I fix it? My docker-compose version is
docker-compose version 1.17.1, build unknown
docker-py version: 2.5.1
CPython version: 2.7.17
OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
while docker version is
Client:
Version: 19.03.6
API version: 1.40
Go version: go1.12.17
Git commit: 369ce74a3c
Built: Fri Dec 18 12:21:44 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.6
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: 369ce74a3c
Built: Thu Dec 10 13:23:49 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.3-0ubuntu1~18.04.2
GitCommit:
runc:
Version: spec: 1.0.1-dev
GitCommit:
docker-init:
Version: 0.18.0
GitCommit:
Thanks in advance,
Flavio
Thaaaaanks, i have been having issues with running airflow and now it worked!!! Ill now be able to automate tasks and be lazier lol,
Let’s gooooo
Great job boy. keep it up.❤
Thanks , it worked Man !
Thanks Marc ! Great work
My pleasure!
Thanks Marc, It helped me a lot!
I love this!!! thanks man!
Great tutorial Old but relevant. Thanks! Marc, I am using Visual Studio Code and everytime I want to save my dag file, I need click a button "Retry as Sudo". Can you tell me what to do here... it is quite annyoing! Regards!
where do I need to write the command at 2:07?
Awesome Marc, thanks for sharing
neat & clean! thanks!!
Thank you ❤️
Awesome man!
Hi Marc, great tutorial. Airflow is running w/o Problems. I tried to use vs code with airflow and found your new video "Configure VS Code to Develop Airflow DAGs with Docker at ease!" However, I don't understand where the Dockerfile come into the picture. Can you please elaborate! ---> Reopen in Container looks totally different as in your video. Thanks
Question: what's the recommended way to increase the number of celery workers using docker compose? Say from 2 workers to 10 workers? Copy&paste worker keys in docker compose yaml files?
No, use docker-compose up --scale airflow-worker=10 :)
@@MarcLamberti Thank you for your quick reply! TIL docker-compose up -scale!
The container for the webserver seems to be restarting continuously every minute or so. Any idea why this may happen?
You must increase the RAM of docker and that is how it will work
Is this usable in production? Could you create a production setup?
super
amazing job
thank u!
Great! but where can i locate the requirements.txt to add for example the apache-airflow-providers-snowflake?
Thank you VERY MUCH!! Marc. This video is very useful.
I finally got it up and running! Thank you, Marc!
how do you create a celery executor if you haven't specified a dedicated backend mysql or Postgres db in the yaml file?
Good job.
Hi, I'm facing the issue PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/20201-03-16' when running "docker-compose up airflow-init" .
Any idea? Thanks
you can run the command with sudo (sudo docker-compose up airflow-init) but i want to know how to do without the sudo
@@DaniloPako Hi, I just solved it without using sudo. You just have to make sure all of the files you create like dags, logs, plugins and the folder you're currently in, the owner and the groupid of them are the user you've log in.
super video Marc je m'abonne !
🙌🙌🙌🙌
I always try to find a docker image to perform experiments.
Thanks for providing a reference that I can refer anytime in future.
Here it is 😁
I have many problems using PythonVirtualenvOperator or ExternalPythonOperator inside docker because you must include system site packages as True (it creates conflicts between venv and base python libraries) or otherwise you will get "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv"
I get the error with URL :
curl: Failed to extract a sensible filename from the URL to use for storage
Is there available docker-compose with mysql?
Can u please share link if you have
I am importing by from airflow import DAG in a file inside my directory but vscode is unable to recognise airflow
That’s because Airflow runs in Docker. You need to connect your VSCode to Docker
Thanks Marc for this video. Question: Do I need to run everytime I spin the containers?
Hi Marc, thanks for your sharing! I'd like to know how to install third-party modules. When I installed yfinance module, there was a dag import error : no module named yfinance.
How can we add and setup airflow.cfg file inside project folder?
this is so much fun and informative, thanks.
Thanks for the wanderful tutorial. I understood that the DAG file I stored in DAG folder will be added to Airflow. But what happen if Airflow is running in remote docker that I only have web access? I can upload DAG from my local disk to remote? Or is there other way to do it?
This instruction really helps me thank you so much !
Thank you so much!
Thanks marc to sort the installation part of airflow👏👏👏👏
Hi I'm a noob I'm using the same YAML file but after running the command "docker-compose up airflow-init" on my ubuntu machine I'm getting this error please help.
ERROR: The Compose file './docker-compose.yaml' is invalid because:
Invalid top-level property "x-airflow-common". Valid top-level sections for this Compose file are: services, version, networks, volumes, and extensions starting with "x-".
You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see docs.docker.com/compose/compose-file/
services.airflow-init.depends_on contains an invalid type, it should be an array
services.airflow-scheduler.depends_on contains an invalid type, it should be an array
services.airflow-webserver.depends_on contains an invalid type, it should be an array
services.airflow-worker.depends_on contains an invalid type, it should be an array
services.flower.depends_on contains an invalid type, it should be an array
Thank you for the awesome tutorial. I do have one question though: how can I install python packages with docker-compose when creating the containers? for example I would like to install Pymongo.
Hi, I would recommend use PythonVirtualenvOperator
@@ramsescoraspe yes, but how do I install a python library like PyMongo, or OpenCV in the container? PythonVirtualenvOperator allows for functions/methods to be created including the module imports they need and then they are destroyed, but I do not have those modules installed in the container. Until now, each time I installed Python modules in containers I did it with a help of a Dockerfile (e.g. inside the Dockerfile I enter "RUN pip install opencv-python") but it is not clear to me how to do the same using a docker-compose.yaml file.
Edit: figured it out: I had to add a pointer to the Dockerfile in the docker-compose.yaml
@@derzemel hi, how do you do this ? (add a pointer)
@ in my case, the airflow webserver service is build like this (the Dockerfile is in the same dir as the compose):
airflow-webserver:
build:
context: .
dockerfile: Dockerfile
@@derzemel do you know how can I add Airflow dependencies inside the docker-compose.yaml file? Also is there a way to provide access to my AWS resources, such as S3, either on the yaml file on the Airflow UI?
anyone know why I can't access the installed airflow docker-compose in ec2 instance via browser?
I have installed airflow using docker-compose in ec2 instance, all containers running, I have set inbound rules security group TCP 8080 port to be accessible. But when I open ec2dnsaddress:8080 on the browser, it shows This site can't be reached.
I have check it also in docker-compose logs airflow-webserver, it doesn't capture access from the outside and it only logs healthcheck
6:32 [11077] Failed to execute script docker-compose
Thank you! very efficiently and clearly explained !
For those that have a Mac and install Docker Desktop, you will not need to install Compose separately. It comes with Docker Desktop
Hi Marc,
I have used docker compose to install airflow.
However, the sample dags seems not to work for me and I found no logs.
getting error : Import "airflow" could not be resolved
while importing 'from airflow import DAG'
Thank you so much, Marc. Great content!
TIL Using YAML aliases and
Love it too 😁
Thank you Marc. I was in hurry to find out how to run airflow and kept failed somehow.
However with your nice clear explanation, nothing is mysterious anymore~
It was great video thanks.
How would i push my custom airflow python file into docker container?
Hi Mark! could I perform these steps without problem on a raspberry pi?
Awesome! Thanks for the tutorial, Marc!
I got all of this running, but once I add a new py in dags it doesnt show on the airflow interface. Anyone had the same issue?
I am getting airflow module not found error when I installed on windows. Do I need to have to do anything else for windows?
I think that airflow doesn’t work on windows
@@matheusaranha278 It works..
Localhost:8080 aint opening for me. How to check the logs for any issues?
VERY EASY TO UNDERSTAND
Hello guys.
Just a tip for everyone:
Do not try to create the airflow folder outside your folder user... You will run into permissions problems (a tried to create a Airflow folder o /opt/, but i strong don't recommend that).
Kindly demonstrate on Teradata and keycloak containerisation
Great video Marc. It's sort of crazy how easy that was (even on WSL2)... Thank you
Some things I'm still considering afterwards:
1) Is this enough for a production deployment of Airflow if the database was decoupled from the rest of the container? If the container crashed for whatever reason all of the connections would be lost, so separating is a good idea.
3) For local testing/debugging of an instance I'm going to try and mount DAGS that exist in another project folder instead of the one that we created.
3b) For local testing I might also try and store connection details in environment variables in the .env file rather than relying on the persistence of the database.
Docker compose is not enough for production, but you can take the same components (containerized) and use kubernetes to go to production
I had to increase the amount of RAM available to Docker to 6GB for this to work on my Mac. Also had to enable permissions to the folder i worked in with CHMOD.
Hi, could you please let me know how you enabled permissions using CHMOD?
I keep getting the following error: "OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "version": executable file not found in $PATH: unknown"
Hello sir, how can we launch every task of etl in a different container as we do via k8s pod operator to launch every task of dag in a different pod?
The only issue is that I can't import anything to my dag from other folders (not dag folder). I don't know why but I get a Import Error
Can someone explain to me why we are running "docker-compose up airflow-init" and then "docker-compose up"?
Thanks for the video!
Hi, thank you for the tutorial, it is very clear and concise, however im facing an issue in which i can't connect to the airflow app through localhost:8080 (browser says empty response). What i've noticed in the logs is that both redis and postgres starts and immediately shuts down and then restart again and so on. For example redis: "redis_1 | 1:signal-handler (1618756176) Received SIGTERM scheduling shutdown..." and for postgres is: "postgres_1 | 2021-04-16 21:57:28.312 UTC [47] LOG: received fast shutdown request" i don't know is someone has faced this issue before, i'm running docker and docker-compose on mac. Any help would be very appreciated
probably you don't need this now - but for anyone who comes across the same issue, it is possible your docker version needs updating to at least v2. If you have v2 already, you may need to allocate more memory (>=5GB to the Docker Engine)
I don't know why but it is giving me some python error when i am executing docker compose up airflow-init.
Any suggestion ?
Thanks Marc ! awesome tutorial.
Thank you 😁
How can I install required packages to docker or how can I mentioned required packages in .yaml file
Thanks Marc!!! Great Job!
Thanks Alex
Great tutorial! Short and sweet. I followed the exact same steps and checked the containers status, redis and postgres were healthy but airflow-scheduler, flower, worker, webserver and triggerer were unhealthy then I deleted all the containers and repeated all the steps and now I'm getting error as "database "airflow" does not exist". Redis and postgres containers are running without any problem. I would appreciate if you can help me understand the error. Thanks.
hi, did you ever resolve the problem? I am having the same issues. Thanks
Hi guys, I got this problem same as you. I am operating in Windows 10. Instead of applying "echo -e....." command, I created a .env file on same directory as .yaml fileAIRFLOW_UID=50000 in it. Problem was solved!
I followed the steps mentioned here, but getting no response from gunicorn master within 120 seconds and the webserver keeps getting restarted. Can anyone help with any lead here please?
This is an awesome tutorial ! Thanks a lot~
Very cool! Unfortunately, I got an error saying 'port 5555 is already allocated..' but I am pretty sure there is nothing on there. So, not sure what's going on.
You should already have something running on that port. You can change the port in docker compose file for flower
Thank you! Marc, got that fixed!
after docker-compose up airflow-init
ERROR: manifest for apache/airflow:2.5.0.dev0 not found: manifest unknown: manifest unknown
😭
change in docker-compose.yaml, line 47:
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.5.0.dev0}
to
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.3.4}
@@akuelperrito faced the same problem and this works!! Thanks so much T__T
excellent walkthrough,,, Thanks :)
Hi Marc
I installed docker desktop at windows using Ubuntu wsl. I changed the dags directory path in .yaml file to my c:\ drive folder in windows.
When I start web UI, it doesn't pick my dags.py file. what can be the issue.
How do we install providers after installing airflow on docker
For me it does not work. Docker compose is creating path "./local" and I can not access it. Airflow can not read my DAGS. it is very frustrating. I have been installing airflow for 8th time and none of them worked...
Any tips for this issue: AirflowException('Celery command failed on host:
While running Airflow 2 via Docker Compose(Just like the above video),
I am unable to successfully execute DockerOperator tasks.
Can you enlighten with a video reference or doc reference about how to properly configure Airflow Docker compose file or Docker Operator to run tasks
When I got to the docker-compose up airflow-init, I get the following error:
"Python-dotenv could not parse statement starting at line 1
Traceback (most recent call last):
File "docker\api\client.py", line 214, in _retrieve_server_version
File "docker\api\daemon.py", line 181, in version
File "docker\utils\decorators.py", line 46, in inner
"
A few dozen more error lines afterwards, but I can't make it work so far
does not do the magic in Windows. beside the curl command not working, the docker-compose up airflow-init command says: top level object must be a mapping.
When i followed the steps and installing, I am getting the error " manifest file not found". I have seen this error is reported by others as well; I changed the image to 2.0.1 and it worked by later I get an error about old version used.
Amazing! Thank you very much!
Glad you like it!
Hi mark. I bought your course but I got an error trying to run the bash operador that insert data into the user table. I have tried the comand alone in the console and it works but when I used inside my dag in my bash operator I got this error bash command failed. The comand returned a non-zero exit code. I have tried a lot but I still can't found a solution for this
Hi Marc, thanks a lot for this!! :)
pleasure :)
Hello Marc, I have recently installed airflow 2 using docker compose file as suggested in this video. But, when I enhanced the dag with mutiple connections i.e., Gdrive->S3, S3->Snowflake,Snowflake->S3 operations using pyspark and sql scripts, the webserver keeps restarting and at times shows unhealthy. Can you please suggest or advice what could have gone wrong or should I consider increasing docker memory?
where to place my custom Python file into the docker container?
Thanks BTW for good video
Thanks, simple and clear ;)
The first time I run "docker-compose up airflow-init", everything is okay. But after I run "docker-compose down" and then I run "docker-compose up airflow-init" once again, I get the message "container for service "postgres" is unhealthy" and the airflow-init container fails. I have to run "docker-compose up airflow-init" once more time to start airflow-init container.
Does anyone get the same problem like me? Could you give me some advice to avoid this, please? Thanks all!
First of all, thank you so much for this *awesome* video. It is really helpful. I followed this tutorial and was able to access AirFlow seamlessly. But I want to have apache-airflow-providers operators. So, I tried giving them in _PIP_ADDITIONAL_REQUIREMENTS and also building using Dockerfile. But nothing worked and I still see "error: command 'gcc' failed with exit status 1". I changed airflow image to 2.1.2-python3.7 as slim versions don't include extra libraries. But no luck. Could you help me resolve this issue?
Wait... would this work with an existing airflow on my local?
yep, as long as you don't bind the same port :)
Anyone facing problem in getting the logs displayed in the UI? Clicking a task --> Log --> gives me a blank log frame. However, it allows me to download it to my machine.
When i execute "docker-compose up airflow-init
" I am getting ModuleNotFoundError: No module named 'airflow' error.
Facing this issue only with root user
@@dhruvatejapadmam6832 You should avoid using the root user :)