Excellent tutorial, Valentin. Everything explained very clearly and concisely. And I really liked how you showed how the .gitlab-ci.yml evolved and why it evolved that way. Nicely done!
great video! and just to update one thing that is missing nowadays:we also have to configure inside the variables the TLS certificate: DOCKER_TLS_CERTDIR: "". otherwise, docker will be available through port 2736 and not 2735. Well, at least I run into this issue with gitlab and that was how I fixed it :) But thanks for your video! really helped me!
Really appreciate the content and the knowledge that you share with us in this video. It is very helpful and probably I will implement it for my projects. Thank you!
Great It works for me, I was facing issue with pipeline build - getting error like "docker and aws command not found". But now it's going smooth without error. Thanks..!
If you can't get past the AWS CLI Login.. First, check if you're running against a "Protected" branch. If you're not, that is probably the reason. By default when setting the Variable, the "Protected" tagged is checked. As defined by GitLab, Protected variables are only exposed to protected branches or tags. The solution to this is to either include that branch with the "Protected" tag or uncheck the "Protected" tag in the variable. Master branches have a "Protected" tag by default.
I keep failed to follow gitlab official document, but finally I did work from your video! Thanks a lot! looks like gitlab-ci 's deploy image is outdated or something wrong with official doc :/
Great Job. I was able to follow along. I just had to add the variable DOCKER_TLS_CERTDIR. Could you use docker:dind as the base image and install aws-cli in it?
I got some issues running gitlabCI where the company forces to use their private registry for the images (they use jfrog). I would really appreciate if you could make tutorial video for gitlabCI using private registry
Did you include how you specify with gitlab runner is used, where that gitlab runner is running, and how that runner has to be setup? I might have missed it.
I was experiencing error where it's fixed by removing that line when I was using a shared runner. Figured that the docker host is already defined inside the shared runner. Thank you for your further explanation. Cheers!
Hi, Thank you for the video. I have a production and a development account and using one gitlab runner for both. It determines which to build and deploy to depending on the branch. How would I store the AWS Access and and Private Key for both in the same repo and use them in the runner?
Actually I think I found a solution - export AWS_ACCESS_KEY=$DEV_AWS_ACCESS_KEY - export AWS_SECRET_ACCESS_KEY=$DEV_SECRET_ACCESS_KEY Unless there's an easier way
@@quinglover3520 You can set it this way for each job! for job 1: - aws configure set aws_access_key_id $DEV_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $DEV_AWS_SECRET_ACCESS_KEY - aws configure set region $DEV_AWS_DEFAULT_REGION for job2: - aws configure set aws_access_key_id $STAGING_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $STAGING_AWS_SECRET_ACCESS_KEY - aws configure set region $STAGING_AWS_DEFAULT_REGION for job3: - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY - aws configure set region $PROD_AWS_DEFAULT_REGION
Thanks Valentin for this tuto, I am trying to push my docker image using jib, I used the same logic and I added a new step to launch the "mvn clean compile jib:build" but I have a 401 http code, must I combine these two steps, if yes how can I use two images in the same step (docker + maven) ?
Hi Valentin, Thanks for save my time I was looking same thing It solved my issue partially could you please tell me how can this AWS ECR image will deploy on k8s aws cluster automatically with new version everytime?
@@vdespa Thanks for reply I am also doing same however is there any way to setup in pipeline its self so whenever pipeline completed it will deploy new deployment in aws k8s cluster?
Great video and really helpful. Just a concern, we are restricted using AWS Access Keys anywhere and advised to use IAM instance roles on our self runners. It would be really helpful if you can create a video to build and push images using IAM instance profile without using AWS Acces keys. Thanks in advance.
@@vdespa So, basically we've installed a self managed Gitlab runner on our EC2 instance and have installed the same following docs.gitlab.com/runner/install/linux-manually.html and registered the same on our Linux instance following the docs.gitlab.com/runner/register/ Let me know if this helps and any further information required. Thanks for your time.
@@vdespa I figured it out. I had an extra space at the end of one of my keys... doh! All set now. I also bought your Udemy course and I am looking forward to it. Cheers
Hello, i wanted to ask if you have a video or course about following specific use-case: - step 1. deploying multiple microservices as docker-images to AWS ECR within a Gitlab-Pipeline (this video is a good start!) - step 2. publishing specific versions of those docker-images from the AWS ECR to a multi-container elastic-beanstalk instance (maybe with help of a Dockerrun.aws.json file and help of AWS Codebuild ? ) If you have any other good example to deploy multiple separate microservices (separate gitlab-projects) to one elastic-beanstalk instance (preferrable multi-docker-container) this would also be of great help! thx for this great video, and looking forward for any response :)
Hi, unfortunately I don't have further content that deals with your use-case. Try breaking down the problem into smaller steps and let me know where you got stuck.
Hey Valentin, thanks for the great tutorial! Everything works for me apart from the last step which is connecting to AWS ECR. Is this method still valid? I am getting multiple retries and after around 20-30 the pipeline fails. Not really sure how can I TSHOOT the connection between GitLab and AWS? I have created the user in IAM, variables are added in CI/CD section of GitLAb. I am not getting any prompt like: Incorrect password etc.
if you need to name your access and secret keys different on the outside. You can do this in your script before the `aws ecr ...` command export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY=
Nice tutorial. But, I have this error: $ aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY Unable to locate credentials. You can configure credentials by running "aws configure". Error: Cannot perform an interactive login from a non TTY device
hi, your video and explanation is very nice. but when i try the docker build command, it saying "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ". i've tried DOCKER_HOST: tcp://docker:2375, DOCKER_HOST: tcp://localhost:2375 and without DOCKER_HOST
we facing this error how to slove An HTTP Client raised an unhandled exception: Invalid header value b'AWS4-HMAC-SHA256 Credential=AKIAZVRYDK32OW27DJHO /20230509/us-east-1/ecr-public/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=c3164e06ba23f93eeb5677abfcee25d7cca686c71b7003ce5b6c266307394806' Error: Cannot perform an interactive login from a non TTY device
Thanks again Valentine! Guys a quick warning for all. Please copy your variables with one line, do not press enter. Otherwise you'll get sth like ```An HTTP Client raised an unhandled exception: Invalid header value b'AWS4-HMAC-SHA256 Credential=secret-id blah blah /20210424/us-west-2/ecr/aws4_request``` bla bla bla. See, Credential has which crashes our pipeline.
![I.N "HALLUCINATION" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/n5B5q1Hwt_U/mqdefault.jpg)
👉 Want to learn more about CI/CD and DevOps? Check my Gitlab CI online course.
vdespa.com/courses/
My Image is already in ECR, how can I update my Image🤣
@@zembelallan5768 Use the same process as shown here.
Excellent tutorial, Valentin. Everything explained very clearly and concisely. And I really liked how you showed how the .gitlab-ci.yml evolved and why it evolved that way. Nicely done!
Thanks, Anthony!
man! the order you explains is perfect! thank you very much! you dont just start writing, you have an order and excellent way to teach us, thanks
You're welcome 😁
This is amazing. Actually I just bougth your Udemy course... you explain things in a very clear way.. Thanks
Thanks, David!
great video! and just to update one thing that is missing nowadays:we also have to configure inside the variables the TLS certificate: DOCKER_TLS_CERTDIR: "". otherwise, docker will be available through port 2736 and not 2735. Well, at least I run into this issue with gitlab and that was how I fixed it :) But thanks for your video! really helped me!
Thanks for mentioning that.
Its, failing for port 2736, 2735 even i set DOCKER_TLS_CERTDIR: "" in variables. Any suggestion ??
my friend is excellent better than many pied training.
Great to hear that. Thank you!
Really appreciate the content and the knowledge that you share with us in this video. It is very helpful and probably I will implement it for my projects.
Thank you!
Glad it was helpful!
Great It works for me, I was facing issue with pipeline build - getting error like "docker and aws command not found". But now it's going smooth without error. Thanks..!
Thanks for sharing. You are welcome. Thank you for leaving a comment.
If you can't get past the AWS CLI Login..
First, check if you're running against a "Protected" branch. If you're not, that is probably the reason.
By default when setting the Variable, the "Protected" tagged is checked.
As defined by GitLab, Protected variables are only exposed to protected branches or tags.
The solution to this is to either include that branch with the "Protected" tag or uncheck the "Protected" tag in the variable.
Master branches have a "Protected" tag by default.
Thanks for sharing.
excellent one, thanks for sharing , can u share if u have done same with Kaniko in place of docker deamon .
Thanks. No, I did not use Kaniko yet.
Thanks for the explanation
You are welcome. Thank you for leaving a comment.
Please make video on automatically deployment on AWS ECS fargat using gitlab cicd
Will do, thanks for the suggestion.
@@vdespa waiting for it.
Great video waiting for it too
I keep failed to follow gitlab official document, but finally I did work from your video!
Thanks a lot!
looks like gitlab-ci 's deploy image is outdated or something wrong with official doc :/
Glad it helped
Great Job. I was able to follow along. I just had to add the variable DOCKER_TLS_CERTDIR. Could you use docker:dind as the base image and install aws-cli in it?
Thanks a lot for this video!
Glad it was helpful!
cool job bro, thanks for your time
No problem 👍! You are welcome. Thank you for leaving a comment.
you have a the video with a example with docker-compose push to ecr from pipeline gitlab?
Not yet, but it is coming soon. Thanks for your comment.
Thank you, helped a lot :)
Glad it helped!
where did you run the command in gitlab for running the project??
All commands are executed by a Gitlab Runner. Was this your question?
@@vdespa right, but Have you created git-runner in this video.
Thanks
I got some issues running gitlabCI where the company forces to use their private registry for the images (they use jfrog). I would really appreciate if you could make tutorial video for gitlabCI using private registry
Sure, thanks for the suggestion. Why did you have issues pushing to the private registry?
Did you include how you specify with gitlab runner is used, where that gitlab runner is running, and how that runner has to be setup? I might have missed it.
Good point. Any Gitlab CI runner with Docker will do. For this example, I have used the shared runners from GitLab.com
Great video, thx!
Thanks for watching.
Hi Valentin, what's the purpose of setting docker host to tcp://docker:2375? Nice vid btw!
This is needed so that the docker client can connect to the docker deamon. I think I have mentioned this in the video.
I was experiencing error where it's fixed by removing that line when I was using a shared runner. Figured that the docker host is already defined inside the shared runner. Thank you for your further explanation. Cheers!
@@null1564 thanks for letting me know. Was the shared runner from Gitlab or your own infrastructure?
@@vdespa it's from my own infrastructure
Thank you!
You are welcome!
Hi, Thank you for the video. I have a production and a development account and using one gitlab runner for both. It determines which to build and deploy to depending on the branch. How would I store the AWS Access and and Private Key for both in the same repo and use them in the runner?
Actually I think I found a solution
- export AWS_ACCESS_KEY=$DEV_AWS_ACCESS_KEY
- export AWS_SECRET_ACCESS_KEY=$DEV_SECRET_ACCESS_KEY
Unless there's an easier way
@@quinglover3520
You can set it this way for each job!
for job 1:
- aws configure set aws_access_key_id $DEV_AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $DEV_AWS_SECRET_ACCESS_KEY
- aws configure set region $DEV_AWS_DEFAULT_REGION
for job2:
- aws configure set aws_access_key_id $STAGING_AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $STAGING_AWS_SECRET_ACCESS_KEY
- aws configure set region $STAGING_AWS_DEFAULT_REGION
for job3:
- aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY
- aws configure set region $PROD_AWS_DEFAULT_REGION
This is good!!
I am glad it helped.
Is it possible to use aws S3 for pushing Docker images?
No, it is not possible.
Thanks Valentin for this tuto, I am trying to push my docker image using jib, I used the same logic and I added a new step to launch the "mvn clean compile jib:build" but I have a 401 http code, must I combine these two steps, if yes how can I use two images in the same step (docker + maven) ?
401 indicates some missing credentials. Did you define the right variables to ensure you are authenticated?
Thanks brother! Now i need to find how to use the pipeline to grab it from the registry and deploy it to AWS ECS FARGATE.
Nice!! Thanksss
Thank you too!
how to CD GitLab over EKS? Thx for the good starter one.
I am working on a tutorial. Thanks for your comment.
Hi Valentin, Thanks for save my time I was looking same thing It solved my issue partially could you please tell me how can this AWS ECR image will deploy on k8s aws cluster automatically with new version everytime?
nothing really happens automatically as ECR is not directly connected to k8s. Use kubectl set image to update the container version.
@@vdespa Thanks for reply I am also doing same however is there any way to setup in pipeline its self so whenever pipeline completed it will deploy new deployment in aws k8s cluster?
Great video and really helpful.
Just a concern, we are restricted using AWS Access Keys anywhere and advised to use IAM instance roles on our self runners.
It would be really helpful if you can create a video to build and push images using IAM instance profile without using AWS Acces keys.
Thanks in advance.
Thanks. So your runner is running in shell mode and not using Docker? Please clarify.
@@vdespa So, basically we've installed a self managed Gitlab runner on our EC2 instance and have installed the same following docs.gitlab.com/runner/install/linux-manually.html and registered the same on our Linux instance following the docs.gitlab.com/runner/register/
Let me know if this helps and any further information required. Thanks for your time.
@@vipul338 Did you register the runner as a shell runner or are you using Docker?
@@vdespa Registered as shell runner.
Wouldn't this mean that anyone that knows our docker registry can easily use aws erc get-login-password to deploy anything to our registry?
No, it won't work. To use get-login-password you need to provide auth.
Thanks!
You bet!
The link to access the yml file is not working. thks
Thanks for reporting. I have updated the link.
I am getting the following error: Error: Cannot perform an interactive login from a non TTY device. i have setup the correct env variables.
Can you make your project public and share a link here?
@@vdespa I figured it out. I had an extra space at the end of one of my keys... doh! All set now. I also bought your Udemy course and I am looking forward to it. Cheers
@@Hendrix312002 thanks for letting me know.
Hello, i wanted to ask if you have a video or course about following specific use-case:
- step 1. deploying multiple microservices as docker-images to AWS ECR within a Gitlab-Pipeline (this video is a good start!)
- step 2. publishing specific versions of those docker-images from the AWS ECR to a multi-container elastic-beanstalk instance (maybe with help of a Dockerrun.aws.json file and help of AWS Codebuild ? )
If you have any other good example to deploy multiple separate microservices (separate gitlab-projects) to one elastic-beanstalk instance (preferrable multi-docker-container) this would also be of great help!
thx for this great video, and looking forward for any response :)
Hi, unfortunately I don't have further content that deals with your use-case. Try breaking down the problem into smaller steps and let me know where you got stuck.
I am getting this error --> Unable to locate credentials. You can configure credentials by running "aws configure"
Are you running this on the main branch?
How can I create a Windows Image in Gitlab CI and push it to AWS ECR ?
It is the same process. You just need to use one of the Windows base images.
I am getting amazon-linux-extras: command not found .... does anyone face the same problem ?
very good. but this is not covering the deployment part..!
Hey Valentin, thanks for the great tutorial! Everything works for me apart from the last step which is connecting to AWS ECR. Is this method still valid? I am getting multiple retries and after around 20-30 the pipeline fails. Not really sure how can I TSHOOT the connection between GitLab and AWS? I have created the user in IAM, variables are added in CI/CD section of GitLAb. I am not getting any prompt like: Incorrect password etc.
Hey, the method is still valid. What is the error you're getting?
@@patrykbambo7247 What do you mean it is timing out? After 60 minutes or what?
$(aws ecr get-login --no-include-email --region ) returns command, then executes
Thanks for sharing. What is the advantage for using this?
if you need to name your access and secret keys different on the outside. You can do this in your script before the `aws ecr ...` command
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
Thanks for sharing.
Nice tutorial. But, I have this error:
$ aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
Unable to locate credentials. You can configure credentials by running "aws configure".
Error: Cannot perform an interactive login from a non TTY device
Make sure you have defined the AWS credentials as environment variables.
@@vdespa Done.
@@vdespa Hi. I set up variables but still have same issue
@@aleksey9442 If you are in a branch, make sure the variables are not protected.
Why don't you safe API id and API secret in GitLab secrets and use it?
hi, your video and explanation is very nice. but when i try the docker build command, it saying "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ". i've tried DOCKER_HOST: tcp://docker:2375, DOCKER_HOST: tcp://localhost:2375 and without DOCKER_HOST
Hey, did you ever solve this?
we facing this error how to slove
An HTTP Client raised an unhandled exception: Invalid header value b'AWS4-HMAC-SHA256 Credential=AKIAZVRYDK32OW27DJHO
/20230509/us-east-1/ecr-public/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=c3164e06ba23f93eeb5677abfcee25d7cca686c71b7003ce5b6c266307394806'
Error: Cannot perform an interactive login from a non TTY device
It it hard to tell just from this error, but make sure that your AWS credentials are correctly set up.
Thanks again Valentine! Guys a quick warning for all. Please copy your variables with one line, do not press enter. Otherwise you'll get sth like ```An HTTP Client raised an unhandled exception: Invalid header value b'AWS4-HMAC-SHA256 Credential=secret-id blah blah
/20210424/us-west-2/ecr/aws4_request``` bla bla bla. See, Credential has
which crashes our pipeline.
Thanks for sharing this!
@@vdespa You're welcome Valentin thank you :)