I think your premise missed an important component. You seem to assert that 'at the top you have one environment'. And then conclude that the main thing terragrunt does is helps you not mess up your state files. At the top, I have MANY environments. And I need a way to consistently test infrastructure against one, and then 'deploy' it to another. This is what terragrunt does for me: I can deploy 'infrastructure v1.3.2' to dev, verify it does what I expected the change to do, and then deploy that version to another environment. Without terragrunt, I have to manually implement the change in infrastructure, which leads to errors.
Hi, my first impression after watching the video was: I'm not using Kubernetes or ECS because I am happy with Docker Compose I'm sorry but you're advising to avoid using Terragrunt only on the basis that Terraform state file management has improved but that is not the only problem that Terragrunt solves. Probably it makes sense not to use it in small environments but in complex environments, it makes life a lot easier, a benefit that is not really noticed until you have dealt with a big or complex environment. Furthermore, the benefits of Terragrunt are hard to be seen at the early stages as the infra part might be looking simplistic and most people will be demotivated by the 'yet another tool that I have to deal with' paradigm Wrapping terraform in a bunch of bash scripts or makefiles with or without Docker takes time, usually, a lot of time when you have to deal either with single or multiple terraform repos and module repos and it turns collaboration and contribution into a policing model. Here is my conclusion based on 7 years of Terraforming and who knows how many projects and Terraform repos: If you have a small single-region infra (even multi-region but it has to be small) say dev/test/prod and a small team where that team owns the whole infra you probably won't see the benefits of Terragrunt and probably you don't need it. Even in this case, Terragrunt will provide a small benefit when it comes to DRY and will help you standardize your infra once you get more experience with it. As the complexity of your environment grows, say something using AWS Control Tower/landing zones with isolated management, security, build or shared services dedicated accounts, or distributed ownership on parts of your infra where segregation of state files really helps, or if you think of an abstract layer approach consolidating your resources into atomic components, Terragrunt will help you simplify that structure improving the overall operational time.
It's not until AFTER you (repeatedly) have to deploy changes from dev infrastructure to prod, by copy/pasting, and run into 'ooh forgot to modify that key', 'oops, missed that value', that you realize you need a different way to deploy. When you have to manually re-code each of your environments, those deployments aren't actually 'tested'. also, re: terraform cloud: it's expensive and the variables aren't versioned, which is a major issue.
I have not yet used Terrargrunt, but I'm about to because there is no choice for me presently when it comes to needing seperate state files and running them in parallel. I'm using 0.13.5 and I don't see how terraform 0.14 without Terragrunt resolves the problem of one massive state file - when it breaks with a big set of infra, its a truly truly horrible thing to have to resolve. And if you have to import resources, it's also difficult. I used to role that way, but never again. This problem has not gone has it? Going back to bash script and modules with seperate state file, solves that problem excellently, it's much easier to fix a problem - but it's slow, and not parallel. So Terragrunt appears to be the only thing that solves this problem. I think at large scale, Terragrunt appears to be a much needed solution.
Thanks for the feedback Andrew. I've received similar comments from others about the need to break up large deployments into separate state files. It's not something I've encountered often, and clearly I need to do more testing. If you have a public example of what you're doing, I'd love to check it out!
I've been using terragrunt since my last post, so this is still a WIP - github.com/firehawkvfx/firehawk-render-cluster The big plusses so far have been: - It's easier to recover corrupted state. Actually it seems less likely in the first place now, and its easier to determine causes for corrupted state. - Automating S3 backend is excellent. - Its easier to implement modules without repetition in some monster terraform root file. This takes getting used to, but it is a better workflow. -It's easy to deploy and destroy a single module without potentially altering the whole infra (and risk destroying something via a dependency trigger as easily). This is a huge benefit! - Encourages better modular design of modules, passing data between modules through data resources/remote state first, and as actual variables as a last resort. Minuses (admittedly could be improved with more user experience): - Sifting through verbose error logs can be tough. The formatting is not so nice. - Running a subset of modules, and perhaps destroying others, in the ways I would like to doesn't seem so straight forward. I might have to revert to using 'count' and variables to turn things on and off. - 'Terraform plan'' seems to be a difficult thing to get working well with lots of dependencies. I tend to always 'apply' or destroy now.
@@FirehawkVFX I have the same problem, we have a code base for four cloud providers, and as well as Kubernetes. We need to be able to have granular state files and terragrunt solves that problem. My company sticks with mostly FOSS solution, so using Terraform Cloud is not an option.
I'm trying to implement terragrunt, I'm having a difficult time finding the benefits other than generating files. Ideally, I'd like to have subdirs of regions that terragrunt would be used to generate files, my modules live in repos. I basically want to use it to speed up generating files so we don't have to copy them into subdirs. My gripe is, I don't want to have my team to support/manage yet another aspect of the infra code.
I have an impression that you never managed huge infrastructure, with mulit regions, mullti envs. TF 0.14 does not change anything here, the only feasible solution is TG or terraspace (too young to use it unfortunately).
You're right Tomasz, I have not managed a multi-region, multi-env solution with Terraform. I'm curious why you feel TG is the best solution? Could you accomplish something similar with workspaces and Terraform Cloud?
So it sounds like you are not a fan of using terraform_remote_state, since by your definition, everything would be in one big state file? Then what used to be pulled from remote state would have to come from variables and the "outer" TF stiches them together?
Multi account terraform is pretty rough, especially when you get into the 20+ account range. Terragrunt makes it easier from a centralized service perspective to maintain those accounts ( ensure security services and inventory management is deployed to those accounts) I manage hundreds of accounts (because that's what aws organizations is pushing) terragrunt solves my concurrency and keeping my terraform dry OOB
@@tonydejesus3774 totally agree. Also TF state is getting bigger and bigger, using TG and small modules makes it very granular and you can only apply on changed parts. Totally different experience.
Before we used to follow issues section in Terraform github repo. Now we also have to follow issues section in Terragrunt github repo. 1+1 = 2 systems (base + wrapper). Sorry what was the thing that we were to solve basically with Terraform?
I disagree with this stament, when you have huge infrastructure with different responsabilitiies you can use a concept of Stacks. And to orchestrate stack you can either use scripts or terragrunt that will define dependencies. Terragrunt benefits is not for states only it more orchestration of infrastructure and blast radius.
I've revised my opinion somewhat since this video came out. For normal deployments I don't think it's necessary, but I can see the benefit for large complex infrastructure builds.
I must admit that I'm new to terraform and terragrunt. So far they both suck. Not even a black hole can match the sucking power of terraform and terragrunt. I may change my mind later. The fact that I can't go into terragrunt console and run find_in_parent_folders() or any function for that matterm to see what it returns, is stupid. I get it, terragrunt console calls terraform console and I can't call a terragrunt function in a terraform console since terraform does not have that built in function. Anyways, debugging code is stupid hard. I come from a background of object oriented programing languages, along with scripting languages . I'm used declaring variables or writing a function and then just calling them out to see the results. Or spitting out the attributes or properties of an object. In terra* world , one cannot just run part of the code. Both of them feel like a boil between the butt cheeks. I get that terraform is a declarative "language" and I'm supposed to tell it I want this, and not care of how it does it, but man it provides no pleasure in writing code. If it were up to me I'd rename them terradumb and terradumber.
That's a charged opinion... but I do feel your overall frustration. Terraform console can help with some of the debugging, but certainly not all of it. If you prefer using a traditional OOP language, then I'd recommend checking out Pulumi. It's using Terraform providers on the backend, but it allows you to use a traditional language like C#, Python, JS, etc. www.pulumi.com/
@@NedintheCloud Or today cdk-tf. I've just used cdk-tf on a new project, and so far it looks really nice. Yeah, it's another layer on top of terraform also, but it gives you so much more than just "keeping your terraform DRY." Also, it's from hashicorp.
when you move up in your career, you'll find it useful. "Terragrunt? Probably not?" The very title of this video is grounds to pass on a candidate in an interview. "additional work that doesn't provide additional value" "if you have terraform cloud, you don't need terragrunt" man... painful.
dude, people are entitled to different opinions and things get very dynamic these days, regarding this issue: since TF 0.14 (we are at 1.5.1, community that is not enterprise) hashicorp resolved many of the issues handled by TG: TF is now DRY for multi-env, multi-region etc, small modules with diff states are possible, state resourcs sharing available, deployment of "versions", etc , lots of progress also on CI/CD on top of terraform that offer scale like TG, so : be open minded always, this guy was not all wrong to think about hashicorp native solutions , ofc the guys that migrated already to TG might want to keep it which is totally applicable
![Digital Circus Episode 4 [TRAILER]](http://i.ytimg.com/vi/abZtsmMUbJo/mqdefault.jpg)
![Kodak Black - Versatile 4 [Official Video]](http://i.ytimg.com/vi/VcP_1Ua__vY/mqdefault.jpg)
I think your premise missed an important component. You seem to assert that 'at the top you have one environment'. And then conclude that the main thing terragrunt does is helps you not mess up your state files.
At the top, I have MANY environments. And I need a way to consistently test infrastructure against one, and then 'deploy' it to another. This is what terragrunt does for me: I can deploy 'infrastructure v1.3.2' to dev, verify it does what I expected the change to do, and then deploy that version to another environment. Without terragrunt, I have to manually implement the change in infrastructure, which leads to errors.
Hi, my first impression after watching the video was: I'm not using Kubernetes or ECS because I am happy with Docker Compose
I'm sorry but you're advising to avoid using Terragrunt only on the basis that Terraform state file management has improved but that is not the only problem that Terragrunt solves. Probably it makes sense not to use it in small environments but in complex environments, it makes life a lot easier, a benefit that is not really noticed until you have dealt with a big or complex environment. Furthermore, the benefits of Terragrunt are hard to be seen at the early stages as the infra part might be looking simplistic and most people will be demotivated by the 'yet another tool that I have to deal with' paradigm
Wrapping terraform in a bunch of bash scripts or makefiles with or without Docker takes time, usually, a lot of time when you have to deal either with single or multiple terraform repos and module repos and it turns collaboration and contribution into a policing model.
Here is my conclusion based on 7 years of Terraforming and who knows how many projects and Terraform repos: If you have a small single-region infra (even multi-region but it has to be small) say dev/test/prod and a small team where that team owns the whole infra you probably won't see the benefits of Terragrunt and probably you don't need it. Even in this case, Terragrunt will provide a small benefit when it comes to DRY and will help you standardize your infra once you get more experience with it.
As the complexity of your environment grows, say something using AWS Control Tower/landing zones with isolated management, security, build or shared services dedicated accounts, or distributed ownership on parts of your infra where segregation of state files really helps, or if you think of an abstract layer approach consolidating your resources into atomic components, Terragrunt will help you simplify that structure improving the overall operational time.
It's not until AFTER you (repeatedly) have to deploy changes from dev infrastructure to prod, by copy/pasting, and run into 'ooh forgot to modify that key', 'oops, missed that value', that you realize you need a different way to deploy. When you have to manually re-code each of your environments, those deployments aren't actually 'tested'.
also, re: terraform cloud: it's expensive and the variables aren't versioned, which is a major issue.
I have not yet used Terrargrunt, but I'm about to because there is no choice for me presently when it comes to needing seperate state files and running them in parallel.
I'm using 0.13.5 and I don't see how terraform 0.14 without Terragrunt resolves the problem of one massive state file - when it breaks with a big set of infra, its a truly truly horrible thing to have to resolve. And if you have to import resources, it's also difficult. I used to role that way, but never again. This problem has not gone has it? Going back to bash script and modules with seperate state file, solves that problem excellently, it's much easier to fix a problem - but it's slow, and not parallel. So Terragrunt appears to be the only thing that solves this problem.
I think at large scale, Terragrunt appears to be a much needed solution.
Thanks for the feedback Andrew. I've received similar comments from others about the need to break up large deployments into separate state files. It's not something I've encountered often, and clearly I need to do more testing. If you have a public example of what you're doing, I'd love to check it out!
I've been using terragrunt since my last post, so this is still a WIP - github.com/firehawkvfx/firehawk-render-cluster
The big plusses so far have been:
- It's easier to recover corrupted state. Actually it seems less likely in the first place now, and its easier to determine causes for corrupted state.
- Automating S3 backend is excellent.
- Its easier to implement modules without repetition in some monster terraform root file. This takes getting used to, but it is a better workflow.
-It's easy to deploy and destroy a single module without potentially altering the whole infra (and risk destroying something via a dependency trigger as easily). This is a huge benefit!
- Encourages better modular design of modules, passing data between modules through data resources/remote state first, and as actual variables as a last resort.
Minuses (admittedly could be improved with more user experience):
- Sifting through verbose error logs can be tough. The formatting is not so nice.
- Running a subset of modules, and perhaps destroying others, in the ways I would like to doesn't seem so straight forward. I might have to revert to using 'count' and variables to turn things on and off.
- 'Terraform plan'' seems to be a difficult thing to get working well with lots of dependencies. I tend to always 'apply' or destroy now.
@@FirehawkVFX I have the same problem, we have a code base for four cloud providers, and as well as Kubernetes. We need to be able to have granular state files and terragrunt solves that problem. My company sticks with mostly FOSS solution, so using Terraform Cloud is not an option.
Honest video. Keep making it. Finally an opinionated speaker in RUclips. God - Thank you for that!
Great video and insights, totally agree!
I'm trying to implement terragrunt, I'm having a difficult time finding the benefits other than generating files. Ideally, I'd like to have subdirs of regions that terragrunt would be used to generate files, my modules live in repos. I basically want to use it to speed up generating files so we don't have to copy them into subdirs. My gripe is, I don't want to have my team to support/manage yet another aspect of the infra code.
I have an impression that you never managed huge infrastructure, with mulit regions, mullti envs. TF 0.14 does not change anything here, the only feasible solution is TG or terraspace (too young to use it unfortunately).
You're right Tomasz, I have not managed a multi-region, multi-env solution with Terraform. I'm curious why you feel TG is the best solution? Could you accomplish something similar with workspaces and Terraform Cloud?
hahaha
So it sounds like you are not a fan of using terraform_remote_state, since by your definition, everything would be in one big state file? Then what used to be pulled from remote state would have to come from variables and the "outer" TF stiches them together?
Multi account terraform is pretty rough, especially when you get into the 20+ account range. Terragrunt makes it easier from a centralized service perspective to maintain those accounts ( ensure security services and inventory management is deployed to those accounts)
I manage hundreds of accounts (because that's what aws organizations is pushing) terragrunt solves my concurrency and keeping my terraform dry OOB
@@tonydejesus3774 totally agree. Also TF state is getting bigger and bigger, using TG and small modules makes it very granular and you can only apply on changed parts. Totally different experience.
Thanks, I totally agree with you, Vanilla Terraform 0.14 can be the solution for most cases while keeping it simple.
Vanilla Terraform 0.14 + shell script is all we need. If one shell script is not enough, just add one more. :)
Before we used to follow issues section in Terraform github repo. Now we also have to follow issues section in Terragrunt github repo. 1+1 = 2 systems (base + wrapper). Sorry what was the thing that we were to solve basically with Terraform?
Super helpful. Thanks!
I disagree with this stament, when you have huge infrastructure with different responsabilitiies you can use a concept of Stacks. And to orchestrate stack you can either use scripts or terragrunt that will define dependencies. Terragrunt benefits is not for states only it more orchestration of infrastructure and blast radius.
I've revised my opinion somewhat since this video came out. For normal deployments I don't think it's necessary, but I can see the benefit for large complex infrastructure builds.
@@NedintheCloud as TF evolved since, huge infra complexity can definitely be handled in native TF
If you have massive infra code, you probably have the resources to have Terragrunt implemented. Otherwise Terraform would be sufficient
Lol, it’s Tuesday year later 😁
I must admit that I'm new to terraform and terragrunt. So far they both suck. Not even a black hole can match the sucking power of terraform and terragrunt. I may change my mind later. The fact that I can't go into terragrunt console and run find_in_parent_folders() or any function for that matterm to see what it returns, is stupid. I get it, terragrunt console calls terraform console and I can't call a terragrunt function in a terraform console since terraform does not have that built in function. Anyways, debugging code is stupid hard. I come from a background of object oriented programing languages, along with scripting languages . I'm used declaring variables or writing a function and then just calling them out to see the results. Or spitting out the attributes or properties of an object. In terra* world , one cannot just run part of the code. Both of them feel like a boil between the butt cheeks. I get that terraform is a declarative "language" and I'm supposed to tell it I want this, and not care of how it does it, but man it provides no pleasure in writing code. If it were up to me I'd rename them terradumb and terradumber.
That's a charged opinion... but I do feel your overall frustration. Terraform console can help with some of the debugging, but certainly not all of it. If you prefer using a traditional OOP language, then I'd recommend checking out Pulumi. It's using Terraform providers on the backend, but it allows you to use a traditional language like C#, Python, JS, etc. www.pulumi.com/
@@NedintheCloud Or today cdk-tf. I've just used cdk-tf on a new project, and so far it looks really nice. Yeah, it's another layer on top of terraform also, but it gives you so much more than just "keeping your terraform DRY." Also, it's from hashicorp.
when you move up in your career, you'll find it useful. "Terragrunt? Probably not?" The very title of this video is grounds to pass on a candidate in an interview.
"additional work that doesn't provide additional value"
"if you have terraform cloud, you don't need terragrunt"
man... painful.
Let me know which company you at so I skip the application
dude, people are entitled to different opinions and things get very dynamic these days, regarding this issue: since TF 0.14 (we are at 1.5.1, community that is not enterprise) hashicorp resolved many of the issues handled by TG: TF is now DRY for multi-env, multi-region etc, small modules with diff states are possible, state resourcs sharing available, deployment of "versions", etc , lots of progress also on CI/CD on top of terraform that offer scale like TG, so : be open minded always, this guy was not all wrong to think about hashicorp native solutions , ofc the guys that migrated already to TG might want to keep it which is totally applicable
@@korenlev9589 my comment was six months ago. reason to pass on your comment.
DRY is a good thing - end of story