Day-8/16 Azure DevOps Terraform Pipeline | Azure DevOps Zero to Hero Full Course | CICD Pipeline
HTML-код
- Опубликовано: 22 авг 2024
- Welcome to Day 8/16 in the Azure DevOps Zero to Hero series. This video will dive deep into infrastructure as code using Terraform and Azure DevOps. We will cover all the relevant concepts while performing a hands-on demo. Our focus will not only be on the Terraform concepts and demos from a beginner's perspective, but we will also implement the end-to-end CICD Pipeline for infrastructure provisioning using Terraform and Azure DevOps.
📌By the end of this video, you will learn the following:
- Introduction to IaC and Tools
- Various Terraform commands and workflow
- Creating Terraform configuration files
- Setting up terraform backend with Azure storage
- Executing Terraform commands using CLI
- Azure DevOps CI Pipeline to init, plan, and archive the plan file
- Azure DevOps CD pipeline to apply the changes
✅𝗔𝗹𝗹 𝘁𝗵𝗲 𝗰𝗼𝘂𝗿𝘀𝗲 𝗼𝘂𝘁𝗹𝗶𝗻𝗲 𝗵𝗮𝘀 𝗯𝗲𝗲𝗻 𝘂𝗽𝗹𝗼𝗮𝗱𝗲𝗱 𝘁𝗼 𝗺𝘆 𝗚𝗶𝘁𝗵𝘂𝗯 𝗿𝗲𝗽𝗼𝘀𝗶𝘁𝗼𝗿𝘆
github.com/piy...
✅ 𝗔𝗹𝗹 𝘁𝗵𝗲 𝗻𝗼𝘁𝗲𝘀 𝗮𝗻𝗱 𝗱𝗶𝗮𝗴𝗿𝗮𝗺𝘀 𝗼𝗳 𝘁𝗵𝗶𝘀 𝘃𝗶𝗱𝗲𝗼 𝗵𝗮𝘃𝗲 𝗯𝗲𝗲𝗻 𝘂𝗽𝗹𝗼𝗮𝗱𝗲𝗱 𝘁𝗼 𝘁𝗵𝗲 𝗗𝗮𝘆8 𝗳𝗼𝗹𝗱𝗲𝗿: github.com/piy...
Please support me and join me in this learning marathon. Your engagement, questions, and feedback will be invaluable as we embark on this journey together.
𝗩𝗜𝗗𝗘𝗢 𝗖𝗛𝗔𝗣𝗧𝗘𝗥𝗦 👀
𝗖𝗢𝗠𝗣𝗟𝗘𝗧𝗘 𝗧𝗨𝗧𝗢𝗥𝗜𝗔𝗟 𝗦𝗘𝗥𝗜𝗘𝗦 👌 It’s 🅵🆁🅴🅴 ⛳
#10WeeksOfCloudOps
• 10WeeksOfCloudOps Chal...
#Azure900WithPiyush
• AZ900WithPiyush - Azur...
Namaste Google Cloud | GCP Tutorial for beginners, a complete playlist for Associate Cloud Engineer Certification Exam:
• Namaste Google Cloud, ...
DevOps Tutorial for Beginners
• DevOps Tutorial for be...
AWS tutorial for beginners
• AWS (Amazon Web Servic...
Azure Data fundamentals DP 900 full course
• Microsoft Azure Data F...
Jenkins Tutorial for Beginners
• Jenkins Tutorial For B...
DevOps and Cloud Podcasts:
• DevOps and Cloud Podcasts
Subscribe to our channel to get notified about the latest videos.
🔥 / techtutorialswithpiyush
-----------------------------------------------------------------------------------------------------------------------
Connect me Here:👇👇
🔥RUclips ▶ / techtutorialswithpiyush
⛳LinkedIn ▶ / piyush-sachdeva
🚀Instagram ▶ / techtutorialswithpiyush
🌍Twitter ▶ / techie_piyush
👨💻 Github ▶ github.com/piy...
✍ Medium.com ▶ / piyush.sachdeva055
📚 dev.to: ▶ dev.to/piyushs...
✨Discord ▶ / discord
-----------------------------------------------------------------------------------------------------------------------
References 📚:
#devops #techtutorialswithpiyush
azure devops terraform pipeline
azure devops tutorial
azure devops pipeline
azure devops full course
azure devops terraform iac
azure devops terraform pipeline yaml
azure devops and terraform
azure devops with terraform
azure devops with terraform and stages
azure devops and terraform tutorial
azure devops terraform tutorial
azure devops ci cd
azure devops zero to hero
azure pipelines
azure devops
azure tutorial
azure devops tutorial for beginners
terraform
Since a long time I am searching for a Azure Devops playlist in RUclips but haven't found or those are very old. Then I found your playlist and it's really very quality content and your way of teaching is also very impressive. Thank you a lot sir!
Thank you for the fantastic feedback. This is for the same reason why I started this series, many of my community members requested this series as there was nothing available free of cost with respect to Azure DevOps. I am glad that you are finding value in this. Thank you
@@TechTutorialswithPiyush yes! Really appreciate your efforts.
The way he explains it is used for real-time implementation. And he explains to the point with logic. I may see other videos of Mr. Piyush . Hats off !!
Thank you so much for the amazing feedback. I really appreciate that.
Great work and hats off. Thanks for the videos.👍🏿👍🏿
Thank you so much 😊🙏
Awesome whoever wanted to see end to end azure devops pipeline please see this video, good work Piyush
This means a lot! Thank you so much!
Thanks a lot for sharing it. I followed the steps and I was also able to run the build pipeline and release pipeline. Keep up the good work!
Well done buddy! keep it up
BRO !!! no bull shit crap talk... no bull shit useless information.... straight clear understandable information delivery in layman language... amazing bro ...
if possible please create a WhatsApp community for learners like us to post queries and help each other sort it out .
Thank you so much bro for the fantastic feedback, that's what keeps me going ❤️
Yes, we already have a discord community where you can ask your questions and queries, help others and interact with like-minded individuals. We also have a weekly call. Link is there in the description section of this video
What a session !!!!! Absolutely loved it!!!!!
Glad you enjoyed it❤️
What an incredible work! congratulations, always impressed when I see people doing such a good work
Thank you so much, truly appreciate your kind words 😊
Thank you for the hard work you put in sir. The video was info packed.
Glad it was helpful!
Great Video, Thank you Piyush
Your video is really helpful.. Thank you for your incredible work...nice explanation...beginners can understand easily....
That was my goal. I recently changed my teaching style and now I am trying to focus more from beginners point of view. I am glad people are appreciating this. Thank you so much!
Holy cow, this helps a lot ! Thank you very much!
Glad to know that :)
Hi Piyush, your videos are very helpful and understandable to everyone.
Please upload remaining videos in the series as quickly as possible
Thank you so much brother for your feedback. Actually I was travelling to India last week and catching up with family and dealing with jetlack. I will be resuming the videos from this week, stay tuned and thanks for your support!
sure bro eagrely waiting for your videos
@@TechTutorialswithPiyush
Thank you sir , for your dedication 4 hours u spent. For the community thank you so much. 😢. Thank you. Again. Really learning great.
Thank you so much , I am glad that you found it helpful 🧡 This is what motivates me to keep pushing the boundaries and keep publishing helpful content. Keep supporting
Great video sir getting many more things learn from you which I can implement in real time 🙏
Glad to hear that, keep up the learning.
Invaluable info. Extremely helpful. Thank you.
Thank you and I'm glad you found it helpful
Thank you so much for this awesome tutorial. One of the best for learning yaml and terraform ! Wishing you more subs and success :)
You're most welcome and I am glad that you found the videos helpful :) Keep supporting and I promise I wont let you down. Many more series in the pipeline.
That's a really wonderful explanation, I m going to watch all videos
Thanks a lot 😊 really appreciate it
Anyone searching for Azure Devops good start plz start from here.
Amazing content with clear explanation. Could you please create a separate Playlist for devsecops for azure devops
Thank you for the amazing feedback. Next video will be about enhancing security in Azure DevOps using advance features. I will try to work on your suggestion. Thank you once again
Really helpful video, thanks for all your handwork.
You're most welcome
@@TechTutorialswithPiyush The satisfaction which i am getting after successfully completion of hands on lab on each videos i can't explain to you, thanks again.
This video is truly incredible! Your detailed explanation of each step from the very beginning is fantastic. Thank you immensely for this valuable content! :)
Thank you for the wonderful feedback, it means a lot
Your teaching is soo clear and simplified Bhaiya. Lobh you for community work you do❤.
Thank you bhai , I am glad you found it helpful
Hi.. Piyush thank you so much for sharing your knowledge. Kindly make a video on terrform workspace concept provisioning an infrastructure for multiple environments like dev, qa, staging....etc
Hello and thank you for the feedback. I already have this item in my todo list which I will work upon once I am done with the ongoing recording of Kubernetes series.
Amazing work Piyush, really like watching your tutorials
Glad you like them! Keep supporting.
Very good video 😊
Thank you Raj!
Great job Piyush, just one request whenever you have any issue please explain in detail the steps of troubleshooting.
Sure Furqan, feedback noted! Also, thank you for the good words.
Please upload all Azure related stuff.. thanks a lot guruji .🎉❤
Definitely brother!
Great explanation Piyush !, Thank you
My pleasure and thank you for the feedback
🙏🙏
Regards,
Doddaka
Thank you!
Awesome, pls make a full azure devops playlist with realtime industry projects.
Thanks Gaurav, this video is part of the Azure DevOps playlist with 16 videos including the realtime projects. Can you please check other videos in the playlist?
Hi Piyush, first of all your videos are really helpful. Thank You for sharing your knowledge with us. I have one question- Can you help me with the script on how to deploy terraform on azure landing zone.
Thank you for the feedback Pragya, truly appreciate your words :)
Regarding the use case, can you please try it yourself and let us know where you are stuck or what issues are you facing? You can join our Discord community (free ofcourse) and share the details, our community will help you out.
Its awsome thanks so much for this video
Glad you liked it!
Great content Piyush. You could possibly just have the init, plan and apply in one build pipeline instead of creating an archive too and then create a release pipeline?
Thank you for the feedback 🙂 Yes, we can do that in one build pipeline as well. The reason why we are doing it that way is so that we can have a separate release stage for each environment in case of multiple environments and we can use quality gates between the environments
Can you please make a video with 3 different environments for dev,test,prod and how to pass parameters using variable group for all 3 environment in a single CI/CD pipeline for creating resources through terraform.
Yes Ayush, I have added this in my to-do list, will create this video soon. Thank you :)
Hi Piyush! This was a great video. I have heard that the tfplan output file can contain secrets. Is this something you should worry about and should that file be encrypted if it is being stored as an artifact?
Thank you! As per my knowledge, you can keep the secret as sensitive, store the tfstate in a remote backend typically a storage bucket/blob storage, restrict access to the bucket with bucket and user policies.
Thank you for such quality contents! it really helped me a lot. would you do tutorial on Automating Microsoft Entra ID using Terraform?
Thank you Jeremy for the wonderful feedback! I truly appreciate your words. Currently I am working on the Kubernetes series and it is taking all my time, I will add this video in my todo list and will work on it afterwards.
@@TechTutorialswithPiyush Appreciate your response! keep it up your good work!
@Jeremyfreedomlover430 Thank you once again ❤️🙏
Excellent series. Thank you!
You're most welcome buddy!
Hi Piyush, Is service principal creation is one time activity or do I need to repeat this task everytime?
usually we create seperate SPNs for seperate teams/environments so there could be multiple SPNs with different permissions but usually a one time thing
Very nice explanation. Some of the command will not work if you are using PowerShell with VS. like : terraform plan | findstr "will be created" instade of grep. Time 34:34.
Hi @piyush,i thing we should use release pipeline for this terraform activity, correct me if i am wrong
Hello, In this project I have used Build and Release pipeline so that you can use the same artifacts to provision infra in multiple environments using the release pipeline.
Hi Piyush,
I tried the lab with a self-hosted agent, but I encountered an error in the last step(release - tf apply). I attempted several solutions to resolve it. In the future, could you please use self-hosted agents for your Azure DevOps labs? This would be helpful, as most customers in real-world environments utilize self-hosted agents.
Thank you!
Hi Mukhesh, What issue did you encounter? Are you sure it is specific to self-hosted agent? Yes, I have used self hosted agent in a couple of videos. I will create more of those
Thanks Piyush for the great videos!! I just had a question if i do all these projects can I put them up on github and apply for devops roles? would this be junior or mid level?
Thank you for the feedback! I started the series keeping in mind to complete Azure DevOps end to end which is basically a SaaS tool but I ended up adding a lot of DevOps concepts as well. Please be advised that you still need to work on other aspects of DevOps such as Monitoring, Alerting, Logging, GitOps, Git branching, Linux administration and so on. But this series covers most of the part and if you complete these projects, may be write some technical blogs and upload it on Github, you can definitely try for an Junior to mid level roles. Good luck!
Thank you for the advice!
@@JJ15982 good luck
Allow me to thank you for producing wonderful content.
I have a question here, In azure devops pipeline how do we authenticate with azure cloud, as I see no env passed for authentication unlike we did in manual process with export commands, did we hardcoded terraform provider section with SP credentials in azure repo?
As per my understanding, service connection serving the purpose of passing env variables to terraform provider section which will provision the infrastructure.
Please help me to understand this.
Obliged for your response in advance!
Hello Sai, Thank you for the fantastic feedback❤️
To answer your question, we use service connection for the authorization. When we were using the tf commands manually, the user that we used to performed az login, had the permissions to list and deploy the resources to Azure. When we are doing it through Azure devops, we create a service connection that creates a service principal which has the permissions to deploy the resources. In a production env, Cloud engineering team is responsible for creating the service principals and assign it certain permissions as per the team and then share it with the respective team and then that team creates a service connection using that service principal.
You can create a service principal from your Azure portal using the cli commands , assign it the contributor role, and then go to your org settings and then create a service connection using the service principal that you have created. Now you can use the service connection in your pipeline.
I hope this answered your question but let me know if you have more questions on this.
@@TechTutorialswithPiyush thanks for responding.
In the manual approach, apart from doing az login we also set env var using the export command(to run terraform commands) in azure devops this is similar to service connection which have the permission to list and deploy our resources in azure.
What about having multiple environments such as devtest, qa and prod. Do we have to have state file for each environment? How will it work for multiple environments?
Yes, you have to create a separate statefile and tfvar file for each environment. You create a folder structure in git repo as well as storage account and store each file in its respective folder
great job bro
Thank you so much 🙏😊
Hi, I have a doubt - I was trying to fetch the template_id of custom role from Microsoft Entra ID - Role and Administration using Terraform Data source azuread_directory_role. I am unable to do that, can you suggest me what I need to do. It is able to fetch the built-in role but not the custom role id.
Hello Shipra, sorry for the delayed response. Can you please share your terraform code so that we can have a look. You can join our discord community and provide details in the az devops help channel.
Hello Piyush! I just wonder if you are using the same resource group for the storage account also? If you are then how can tf destroy let you delete resource group while its still containing the storage account and the tfstate file init. It didn't let me though! It showed me only 4 changes while doing the tf apply and tf destroy.
Hey, The storage account and resource group containing tfstate file was created as a prerequisite and not part of terraform.
Hi Piyush
I have added a condition in azure-pipelines.yml such that master pipeline runs only in CTE & PROD environments. How can I make the azure-pipeline.yaml non-editable in feature branches so that no one can remove the conditions that I have added?
May be you can apply branch policy and restrictions
learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops&tabs=browser
thank you for making this type of video
Glad you liked it, my pleasure!
Hello , Can you please explain again why the Archive file task was added in the CI pipeline
Hello Rehan, I added that for a couple of reasons:
- So that I can show both the CI and CD pipelines.
- So that we create artifacts of any infra changes, we can use the same artifacts to provisioning multiple environments , keep a track of those changes and use that to create exactly same environments. I hope this answered your question.
@Piyush Sachdeva please suggest me trouble shoot for extracting files. I am not able to understand why in CD pipeline files is not extracting the error is specified archive because it cannot be accessed no such file or directory stat...
verify the path specified in your pipeline. looks like it's not able to find the archived build. If you still face the issue, join our discord server and share your pipeline yaml over there to assist you further
Great Piyush 👍
Thank you buddy!
HI Piyush I had follow your steps iam facing one issue while running release pipeline "The process '/opt/hostedtoolcache/terraform/1.8.2/x64/terraform' failed with exit code 1"
Hello, This is the not the actual error but the error by terraform itself. Many people have faced this issue and it was due to incorrect directory, Can you please check the directory path where you have extracted the build artifacts and if that is matching with the directory where you are running the terraform apply command. Here are the issue reference as well
discord.com/channels/1030513521122885642/1201476235746885693
can you plz show how to do from pipeline section instead from build pipeline from repos
Its one and the same thing
Hello I am trying to deploy the resource, but my devops pipeline is not initializing terraform, I am getting error"##[error]Error: There was an error when attempting to execute the process '/opt/hostedtoolcache/terraform/0.14.11/x64/terraform'. This may indicate the process failed to start. Error: spawn /opt/hostedtoolcache/terraform/1.0.0/x64/terraform ENOENT
Finishing: terraform init" I appreciate your suggestion
Hello,
This is not the exact error. Let's do one thing, join our discord server and share your details in #help-az-devops channel, provide the complete error logs along with your pipeline code and someone will assist you.
thecloudopscommunity.org , click on join us to join the discord server
@@TechTutorialswithPiyush Thank you I have posted in the community channel
Hey Piyush, I couldn’t find the archive extension and artifacts in market place? Any suggestions? I am using my own windows build agent. Not sure if that is causing an issue. At 1.01 you entered archive. I can’t see it ?
Hi, It should not be impacted by the build agents, can you join our discord server and share the steps you have followed? there is a dedicated help azure devops channel which you can use.
Hi thanksfor the video, can you do a video of an API gateway management using azure devops pipeline.
Thank you for the feedback. Azure DevOps series has now been completed with the topics from beginners point of view but I will try to create this video outside the playlist.
@@TechTutorialswithPiyush again thankyou for all your effort you kept for creating all these videos. And very usefull to everyone who wants to learn.
@@kavacham222 These words means a lot to me, thank you for your support :)
error at tf init task, not able to connect to storage account!
Error: Failed to get existing workspaces: containers.Client#ListBlobs: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailure" Message="This request is not authorized to perform this operation
Hello, Please share your pipeline code. Looks like you dont have permissions to run az commands. Feel free to create a thread in our discord channel with all the details and someone will help you
Hi, Thaks for the video. When I change the includeRootFolder to false the build pipeline fails. Still trying to de bug
Hello Fahad, what error are you getting when you use includeRootFolder to false? Can you join our discord community and share your pipeline code + error details. Also, what all files do you see in archive?
Hi Piyush, yes sure I will share it in the discord group. Thanks for your reply
Sir I have completed MCA in after that 2021 September after that I was trying to crack some Government fail cracking that. In 2022 November i Started practicing azure Devops with my frd who is working in devops field. As completed one certificate az-900 & az-104 will complete till December 29 sir I need your help in getting job. how I can get job because I am fresh I am not getting job guide me sir for getting job.
Glad to know that you have completed these certifications and working hard to transition into IT. I would suggest doing the hands-on for anything you are learning. Try to implement different cloud and DevOps use cases and upload the projects on Github and write technical blogs on sites such as medium.com. Create the tailored resume and include the complex projects you have built. Let me know if you have more questions.
@@TechTutorialswithPiyush thank you Sir
@piyush I have done several projects and I have also written several blogs, my thing here is I am finding it difficult to upload those project to my GitHub, can you help me sir?🙏🙏
In azure devops during build process at terraform plan task while performing "acquring state lock". it get stuck i waited for ages like 1hr but it still showing acquring state lock. i think it has issue could you please tell me what it is and how to fix
Try to run it manually and see what issues you are facing? Either its not able to acquire lock on statefile, or it doesnt exists. I'd suggest runnning all the commands manually first
Where is next video no. 9?
sent that for editing, should be published soon
why are we archieving and publishing,? rather than directly publishing the artifact
We have to bundle the artifact into a package before publishing it, we cannot publish the folder directly
HI @piyush
can you please plan to share AWS devops Videos paly list as well ..
Thanks.
I will try to create in near future
What to do with parellel job error??
You have to fill the form using the link given in the error message
Sir when the new video is coming?
There has been a slight delay as I was travelling to India and catching up with my family. will resume the recording from day after tomorrow and hoping to publish it soon
Hi, if I create this in west europe will it be okay??????
That should be okay but you can use the location nearest to you for low latency and low cost network transfers
Hi , Till when we can accept this playlist to get completed ?
I'm doing my level best sometimes when I'm facing some issues, then there is the delay in fixing those errors. I'm travelling to India today so next video will be around the weekend
@@TechTutorialswithPiyush Please try to finish as soon as possible. It will break the flow for us. If delay in videos
@@Sid27_12 I understand but you need to have some patience. It takes me 5-6 hours creating a single video and when I face some issues it could take days to create the single video. it takes reaearch, recording, editing, publishing and a lot of troubleshooting in demos. please have some patience.
❤❤❤
Thank you sir
can you make one zero touch pipline create with yml file for any sample project
To make the one touch pipeline, you just have to remove the pre deployment approval gate from the release pipeline and rest everything is zero touch. In most of the organizations, they have an approval step or a quality gate before the production deployment that is why I have included it.
Error: There was an error when attempting to execute the process '/opt/hostedtoolcache/terraform/1.9.2/x64/terraform'. This may indicate the process failed to start. Error: spawn /opt/hostedtoolcache/terraform/1.9.2/x64/terraform ENOENT. Getting this error in release pipeline can anyone help
This is not the actually error. It simply says that terraform command failed. If you are getting this error while the terraform apply stage, it could be due to the wrong directory. Make sure you are running the tf apply in the same directory where you have extracted the build. If you still face the issue, join our discord server and share the pipeline code and complete log in the #help-az-devops channel.
@@TechTutorialswithPiyush thanks for your reply, yes the tf init in release stage was getting initiated in an empty directory. so what I did was add the apply task stage in the yml code of build pipeline itself with the condition tf plan succeed. it did work and deployed the resources but I am not sure if it is the best practice
@@vatsalsingh1054 That is also okay but what you can do. When you extract the files in release pipeline, it creates another folder inside the extracted folder, so updated your release pipeline to add that build folder path in the tf apply state
getting this below error at release pipeline state - 2024-05-02T22:12:13.7423434Z ##[error]Error: There was an error when attempting to execute the process
'/opt/hostedtoolcache/terraform/1.8.2/x64/terraform'. This may indicate the process failed to start.
Error: spawn /opt/hostedtoolcache/terraform/1.8.2/x64/terraform ENOENT
This is not the actual error. I believe you are running the terraform apply to a different directly than where you have extracted the build artifacts. Please verify that.
Someone else has also faced this issue in our discord server, if you still face the issue, join the server and ask the question in the az-devops help channel.
Hi.Had the same error. In my case in Extract files task destination folder was '$(System.DefaultWorkingDirectory)'
and in Terraform: init task configuration directory was '$(System.DefaultWorkingDirectory)/'.
So the issue was in missing slash sign.
Hi Piyush,
I tried to implemnt it the First Part was successful but getting the 2023-12-10T04:12:32.1699388Z ##[error]Error: The process '/home/ubuntu/_work/_tool/terraform/1.6.5/x64/terraform' failed with exit code 1 when doing Azure DevOps Terraform I changed the build pipeline that fixed for you but Im still facing the same issue after doing that as well
I would suggest joining our discord and sharing the error details along with your pipeline code to assist you further
Hi Piyush..Thanks for sharing this videos its really helpful :) Need your help same all tasks I'm performed but facing error when release pipeline triggerd...its error "ArchiveFiles" error]Specified archive: /home/vsts/work/r1/a/_Privatelab/27- can not be extracted because it can not be accessed: Error: ENOENT: no such file or directory, stat '/home/vsts/work/r1/a/_Privatelab/27-'
at Object.statSync (node:fs:1583:3)
Can you please help me on this error...Thanks in advance..
one more error also The process '/opt/hostedtoolcache/terraform/1.6.5/x64/terraform' failed with exit code 1
Thank you for the feedback, looks like it's not able to find the files. can you ensure you are using the right directory path? also, can you share the error details along with the pipeline code , someone will help you