Love these DevOps redpills! Actually learnt a lot more practical stuff and good practices in this 5min video than in longer tutorial using tasks and stuff! Subscribed
Glad to hear! Out of curiosity, what's in the 5 minute videos that's not in the longer tutorials? Sorry for super late response. I just discovered this comments UI for publishers and am finding so many I missed - when I didn't respond immediately to a notification.
@@JulieNgTech You’re amazing! I love you so much. You’re a true gift to anyone who wants to learn. You’re admirable in every way! I hope I get to meet you in person some day. I always tell people I know who are starting out in enterprise cloud services about you. Enjoy your day!
Thanks for this Julie! I created the local backend conf file outside of the git and TF working dir all together so I don't even need to worry about having to add my local stuff in gitignore. And wrote a simple bash script wrapper to execute the backend conf file and export local TF_Var's all in one shot. :)
Really interesting. I liked the reasoning for using CLI instead of tasks. I would love to see a small video on how to store the state file :') And how to use a service connection with Bash :')
wow, this video is a masterpiece, all the questions I needed answers to wrapped up in just over five minutes. Thank you, Julie! Heading over to the blog and gonna implement this stuff right away.
Great content, It would be great to have a 2nd edition to show the good practice with the CICD pipeline, Complete deployment and Approvals before deploying
Hi Roy, yes, I have that on my todo list to walkthrough the pipelines in this repo github.com/azure/devops-governance which does pull requests, deployments, etc. I also recently discovered I don't like how they are done and want to make the git workflow easier and just put a "manual approval" on the service connection. I forget exactly what, but I found a problem that I could not easily get into production without going through all the git checks. In hindsight, I would undo lots of that 😅 and sacrifice some automated approvals for faster deployments and just have 1 manual approval step. Would you want to see this as a video?
@@JulieNgTech good question, i did not compared the time. But i will. Currently a project where i have to use parameterized custom script extensions with windows powershell. A horror, 6 hours bugfixing. Although windows user .\ as current path in the terraform config for uploaded script you have to use ./ - lmao
Hi @@JulieNgTech. I had a look at the devops-governance repo used in some of your examples and I quite like how the drift detection is set up there. A video on how to set that up with a configured response might be cool?
@@picklednewtons6282 added request to my list :) Be aware, I need to rework the governance repo. Azure DevOps gives least permissions when you have multiple group assignments (unfortunately a footnote in the docs), so the I need to add an additional AAD group per business unit, because ARM uses additive permissions. That is also on my todo list for early March.
Amazing knowledge.Thanks for putting things into perspective. Would be really helpful if you point us to a Repo where you are implemented all this please. Waiting for your series on AzureDevops with Terraform if possible please. Cant thank you enough.You Rock!!
Thanks Kiran! You can look at some examples here github.com/Azure/devops-governance/tree/main/azure-pipelines Let me know if there's a topic you're interested that's not in the repo - or poorly documented in the repo. Happy deploying!
Those are 2 separate event triggers. The first you want is for the PR trigger like in this example, which only does `init` and `plan`. github.com/Azure/devops-governance/blob/main/azure-pipelines/pull-request.yaml Theoretically the `apply` would be a different pipeline that triggers on push to the target branch of the PR, e.g. `main`. So commits to you e.g. `feat/*` branches trigger the PR pipeline. Once someone merges it into `main` or whatever you naming convention is, the pipeline would do an `apply` BUT, a BIG HUGE BUTTTTT the Terraform plan that was approved in the PR would be stale. The 2nd pipeline would run a second plan and apply potentially without human intervention. That's why I don't put Terraform apply in the pipelines. Too risky for me.
Excellent, super useful, thanks so much for sharing. Question on best practices for managing TF state of multiple envs (dev, qa, prod etc). Do you store each env state in a diff storage account, or have a centralised "devops" storage account where you'd have state for each env in the same path, but distinguished by suffix (just like using tf workspaces)? Thanks! Oh, and would love more content, but appreciate super time consuming to put together, love your insights)
Hi Jon-Paul, thanks for the feedback. Re: Terraform state, it depends on your requirement. The biggest thing to keep in mind that the storage account is your security boundary. In a high trust scenario, you can use a single storage account for various state files. But for lower trust scenarios, you want to split them up, not just to prevent read access, but also listing the containers and files in the storage account. See this doc for a more technical detail github.com/julie-ng/cloudkube-aks-clusters/tree/main/backends Hope that helps!
first step of debugging is to compare configuration, e.g. ARM credentials, state files, terraform versions, etc. check your error messages and some googling should help you ;-)
I got error when running terraform init -backend-config=azure.conf: Terraform initialized in an empty directory! However I did have a main.tf in the current folder.
When you say "current folder", did you change it with cd? If I remember correctly that doesn't work for various reasons. Your code is running in a sub shell. Instead use the `workingDirectory` property in your script. See this doc for details: docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#script
What an amazing video! Really informative and touching on fundamental points. I have a question related to Terraform State Lock. When, say, a CI build is canceled in the middle of a `terraform apply`, the state gets locked. Could there be a "post run" task that would perform some sort of "unlock" of the state automatically? Or, is it the kind of thing that needs to have a human in the loop to check against possible infrastructure changes before unlocking the state?
Good question. Instead of asking how, I would ask *why*. Why would a terraform apply need to be cancelled? How often would that occur? The answers to those questions should help you decide whether having that post run task makes sense. The terraform lock feature exists for a reason ;-) In my case, needing to manually unlock it is a rare exception, which is why I would not include it in any of my pipelines. What's your scenario?
@@JulieNgTech Great perspective (pointing to the *whys* instead of *hows*). In general, I would like to be able to deliberately cancel a CI job when it is not relevant anymore (e.g.: it is targeting an out-of-date commit), to save time and resources. This scenario usually implies running the CI with the latest pushed commit right after cancelling the ongoing one. Of course, it is not an ideal scenario because, in the first place, one should not be mindlessly pushing commits. Nevertheless, I have observed it happenning in practice with some frequency. Under these circumstances, I would argue that having a post-run job is beneficial. Given that the purpose of the state lock is to prevent multiple, concurrent writers from corrupting the state, and we know for sure that the ongoing apply was cancelled (and will therefore not write to the state anymore), removing the lock seems like a direct logical conclusion. Wdyt? Thanks for leading me towards this reasoning btw :)
Great video. Any tips on Terraform credentials in AWS? storing them as parameters in Azure DevOps seems convenient but not sure how auditable or safe that would be..
Not that I know of - because I work for an AWS competitor ;-). But great question! Sorry for the wicked late response. But I put your question in my latest video in the AMA part. For the auditable, etc. definitely go back to your org's security folks and ask them what their requirements are. Then you can see if/how they map to Azure features. We use ADO internally and our credentials are integrated. So it should be possible for you too :)
If you were using different folders for different types of resources, network folder for virtual networks and subnets, application folder for data factories, databricks, etc .. how would you go about then having a file to be able to access the terraform state of those other resources when states are managed by resource type as well? The terraform_remote_state provider needs the config block which asks for the same stuff in your azure.conf?
Hey Hyn, why would you put networks, databricks, etc. in their own folder? If they need the access to the same Terraform state file, wouldn't they be a single Terraform IaC deployment?
@@JulieNgTech The reason i've done it is to separate out resources by a type so as to limit the possibility of a random network change causing a databricks rebuild or vice versa. I also wouldn't want to chug through the entire environment with an apply if all I was doing was adding a subnet. I also think I saw it on TF best practices.
@@Klainn avoiding unintentional builds is one of the most challenging practices to master in DevOps and in my opinion a life-long journey. For infra, I sometimes still flinch when pushing. That being said, I recently gave a talk at DevOps.js that talked about triggers in git repos (one vs many) and that might help you ruclips.net/video/VAsUutOq9mY/видео.html
IMO Best Practice is to use 2 Storage Accounts, 1 for production and 1 for non-production - both using SAS tokens to access the state file. This follows cloud governance best practices to separate RBAC and thus credentials for production. Unfortunately you need 2 storage accounts and it is not enough to scope to an Azure Storage Container because Terraform workspaces will query the entire Storage account to find all statefiles it thinks might be a "workspace." Does that soundbyte answer your question? I've been meaning to blog or do a video about it with a demo but I haven't gotten around to it.
Thank you for this! Just what I needed, solved a question I had been struggling to find an answer to.
Love these DevOps redpills! Actually learnt a lot more practical stuff and good practices in this 5min video than in longer tutorial using tasks and stuff! Subscribed
Glad to hear! Out of curiosity, what's in the 5 minute videos that's not in the longer tutorials?
Sorry for super late response. I just discovered this comments UI for publishers and am finding so many I missed - when I didn't respond immediately to a notification.
@@JulieNgTech Most videos are introductory or unneccessary long. Yours was straight to the point and practical.
@@alexandreg3933 Seconded! Thank you Julie.
Best content as like previous comment I totally agree that this 5 min video taught a lot of quality stuff
The "full" article was very helpfull - thank you!
How does this video not have more likes?? Seriously!
Thanks! I'm glad you enjoyed the video that much. I think the likes come over time :)
@@JulieNgTech Not sure if you know this, but your post is becoming a DevOps pipeline standard.
Amazing content, thank you!
Glad you enjoyed it! And I am relieved it is still relevant many years later 😅
Your depth of knowledge has earned you a new subscriber! Keep 'em coming :)
Thanks for the feedback. I need these little bits to find motivation to make more videos.
You're welcome. I found tremendous value in your video and article.
This is brilliant, thanks. Heading over to read the article next
I hope you find the article helpful as well. Let me know if you feel something is still missing.
Because of you , I will pick Azure over AWS
Really? That made my day 😻 I need more comments like this so I can make videos during work hours.
@@JulieNgTech You’re amazing! I love you so much. You’re a true gift to anyone who wants to learn. You’re admirable in every way! I hope I get to meet you in person some day. I always tell people I know who are starting out in enterprise cloud services about you. Enjoy your day!
Super helpful article! Thank you very much, Julie.
You welcome Alexander! Thank you for the feedback ❤️
More of this pls, i just started sys admin role, this stuff is clutch tips.
Thanks for this Julie! I created the local backend conf file outside of the git and TF working dir all together so I don't even need to worry about having to add my local stuff in gitignore. And wrote a simple bash script wrapper to execute the backend conf file and export local TF_Var's all in one shot. :)
Automation and scripting FTW :)
The article is awesome.. very detailed and with reasoning.. very helpful 👌
Really interesting. I liked the reasoning for using CLI instead of tasks. I would love to see a small video on how to store the state file :')
And how to use a service connection with Bash :')
Great tips!!! Saved me a lot of time . Thanks
Awesome! Happy coding for Pipelines :)
Thanks Julie. Very helpful
Glad it was helpful! Let me know what else you'd like to see :)
Very Helpful - Thank you Julie
Glad it was helpful! Thanks for watching! If it's helpful, please consider subscribing for more :)
I really liked your tips and it's short and sweet
Glad you liked it!
Very informative, thanks Julie. I will check out your article 👍
wow, this video is a masterpiece, all the questions I needed answers to wrapped up in just over five minutes. Thank you, Julie! Heading over to the blog and gonna implement this stuff right away.
Thanks Matt, glad it was helpful!
This article was great, really helpful! Exactly what I needed. Thanks!
Glad it was helpful!
Excellent Video and Article.
Thanks !
Glad you liked it!
Great content,
It would be great to have a 2nd edition to show the good practice with the CICD pipeline,
Complete deployment and Approvals before deploying
Hi Roy, yes, I have that on my todo list to walkthrough the pipelines in this repo github.com/azure/devops-governance which does pull requests, deployments, etc. I also recently discovered I don't like how they are done and want to make the git workflow easier and just put a "manual approval" on the service connection. I forget exactly what, but I found a problem that I could not easily get into production without going through all the git checks. In hindsight, I would undo lots of that 😅 and sacrifice some automated approvals for faster deployments and just have 1 manual approval step. Would you want to see this as a video?
Hello@@JulieNgTech yes I would like to see that on video :)
Love it! Thank you for the great info and tips!
Thanks Lorenzo! Thanks for watching subscribing. Let me know what you want to see more of :)
You're a legend
Not my goal to be a legend. It's to teach people (and stop repeating myself LOL)
But if it happens I won't complain ;-)
thanks for the tipps - i used to use the buildin modules too
Did removing them make accelerate your deployment frequencies?
@@JulieNgTech good question, i did not compared the time. But i will.
Currently a project where i have to use parameterized custom script extensions with windows powershell. A horror, 6 hours bugfixing. Although windows user .\ as current path in the terraform config for uploaded script you have to use ./ - lmao
Love this. great content. Thank you
Thanks, good tips!
Great video and write up, thank you Julie.
Thank you for the feedback ♥️ let me know if there are other topics you're interested in and could use a video :)
Hi @@JulieNgTech. I had a look at the devops-governance repo used in some of your examples and I quite like how the drift detection is set up there. A video on how to set that up with a configured response might be cool?
@@picklednewtons6282 added request to my list :) Be aware, I need to rework the governance repo. Azure DevOps gives least permissions when you have multiple group assignments (unfortunately a footnote in the docs), so the I need to add an additional AAD group per business unit, because ARM uses additive permissions. That is also on my todo list for early March.
Amazing knowledge.Thanks for putting things into perspective. Would be really helpful if you point us to a Repo where you are implemented all this please. Waiting for your series on AzureDevops with Terraform if possible please.
Cant thank you enough.You Rock!!
Thanks Kiran! You can look at some examples here
github.com/Azure/devops-governance/tree/main/azure-pipelines
Let me know if there's a topic you're interested that's not in the repo - or poorly documented in the repo. Happy deploying!
Liked, subscribed and did hit that bell! :) Thanks for sharing.
Thank you! Let me know if there's other questions I can help answer :)
So, when you push a PR to Git, how do you get Terraform to run only "init" and "plan" and not "apply" until the PR has been approved?
Those are 2 separate event triggers. The first you want is for the PR trigger like in this example, which only does `init` and `plan`. github.com/Azure/devops-governance/blob/main/azure-pipelines/pull-request.yaml
Theoretically the `apply` would be a different pipeline that triggers on push to the target branch of the PR, e.g. `main`. So commits to you e.g. `feat/*` branches trigger the PR pipeline. Once someone merges it into `main` or whatever you naming convention is, the pipeline would do an `apply` BUT, a BIG HUGE BUTTTTT the Terraform plan that was approved in the PR would be stale. The 2nd pipeline would run a second plan and apply potentially without human intervention.
That's why I don't put Terraform apply in the pipelines. Too risky for me.
Excellent, super useful, thanks so much for sharing. Question on best practices for managing TF state of multiple envs (dev, qa, prod etc). Do you store each env state in a diff storage account, or have a centralised "devops" storage account where you'd have state for each env in the same path, but distinguished by suffix (just like using tf workspaces)? Thanks! Oh, and would love more content, but appreciate super time consuming to put together, love your insights)
Hi Jon-Paul, thanks for the feedback. Re: Terraform state, it depends on your requirement. The biggest thing to keep in mind that the storage account is your security boundary. In a high trust scenario, you can use a single storage account for various state files. But for lower trust scenarios, you want to split them up, not just to prevent read access, but also listing the containers and files in the storage account. See this doc for a more technical detail github.com/julie-ng/cloudkube-aks-clusters/tree/main/backends
Hope that helps!
@@JulieNgTech Thanks so much for the info!
Can you please create a detailed video on creating a yaml file from scratch and how to segregate stages, jobs and steps in different yaml
Why from scratch? Also have you seen this video of mine that does talk about stages, jobs, etc.? ruclips.net/video/e0bF1LlclEs/видео.html
Hello i have a Question when i run terraform project locally it works fine for me but in azure devops environment does not any idea
first step of debugging is to compare configuration, e.g. ARM credentials, state files, terraform versions, etc. check your error messages and some googling should help you ;-)
I got error when running terraform init -backend-config=azure.conf: Terraform initialized in an empty directory! However I did have a main.tf in the current folder.
When you say "current folder", did you change it with cd? If I remember correctly that doesn't work for various reasons. Your code is running in a sub shell. Instead use the `workingDirectory` property in your script. See this doc for details: docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#script
What an amazing video! Really informative and touching on fundamental points.
I have a question related to Terraform State Lock. When, say, a CI build is canceled in the middle of a `terraform apply`, the state gets locked. Could there be a "post run" task that would perform some sort of "unlock" of the state automatically? Or, is it the kind of thing that needs to have a human in the loop to check against possible infrastructure changes before unlocking the state?
Good question. Instead of asking how, I would ask *why*. Why would a terraform apply need to be cancelled? How often would that occur? The answers to those questions should help you decide whether having that post run task makes sense.
The terraform lock feature exists for a reason ;-) In my case, needing to manually unlock it is a rare exception, which is why I would not include it in any of my pipelines. What's your scenario?
@@JulieNgTech Great perspective (pointing to the *whys* instead of *hows*). In general, I would like to be able to deliberately cancel a CI job when it is not relevant anymore (e.g.: it is targeting an out-of-date commit), to save time and resources. This scenario usually implies running the CI with the latest pushed commit right after cancelling the ongoing one.
Of course, it is not an ideal scenario because, in the first place, one should not be mindlessly pushing commits. Nevertheless, I have observed it happenning in practice with some frequency.
Under these circumstances, I would argue that having a post-run job is beneficial. Given that the purpose of the state lock is to prevent multiple, concurrent writers from corrupting the state, and we know for sure that the ongoing apply was cancelled (and will therefore not write to the state anymore), removing the lock seems like a direct logical conclusion. Wdyt?
Thanks for leading me towards this reasoning btw :)
Great video. Any tips on Terraform credentials in AWS? storing them as parameters in Azure DevOps seems convenient but not sure how auditable or safe that would be..
Not that I know of - because I work for an AWS competitor ;-). But great question! Sorry for the wicked late response. But I put your question in my latest video in the AMA part. For the auditable, etc. definitely go back to your org's security folks and ask them what their requirements are. Then you can see if/how they map to Azure features. We use ADO internally and our credentials are integrated. So it should be possible for you too :)
If you were using different folders for different types of resources, network folder for virtual networks and subnets, application folder for data factories, databricks, etc .. how would you go about then having a file to be able to access the terraform state of those other resources when states are managed by resource type as well? The terraform_remote_state provider needs the config block which asks for the same stuff in your azure.conf?
Hey Hyn, why would you put networks, databricks, etc. in their own folder? If they need the access to the same Terraform state file, wouldn't they be a single Terraform IaC deployment?
@@JulieNgTech The reason i've done it is to separate out resources by a type so as to limit the possibility of a random network change causing a databricks rebuild or vice versa. I also wouldn't want to chug through the entire environment with an apply if all I was doing was adding a subnet. I also think I saw it on TF best practices.
@@Klainn avoiding unintentional builds is one of the most challenging practices to master in DevOps and in my opinion a life-long journey. For infra, I sometimes still flinch when pushing.
That being said, I recently gave a talk at DevOps.js that talked about triggers in git repos (one vs many) and that might help you ruclips.net/video/VAsUutOq9mY/видео.html
What about state files from terraform?
IMO Best Practice is to use 2 Storage Accounts, 1 for production and 1 for non-production - both using SAS tokens to access the state file. This follows cloud governance best practices to separate RBAC and thus credentials for production.
Unfortunately you need 2 storage accounts and it is not enough to scope to an Azure Storage Container because Terraform workspaces will query the entire Storage account to find all statefiles it thinks might be a "workspace."
Does that soundbyte answer your question? I've been meaning to blog or do a video about it with a demo but I haven't gotten around to it.