GitOps: Branches, directories, or different repositories for the desired state of environments?
HTML-код
- Опубликовано: 4 окт 2024
- Today, we tackle a common GitOps question: "Should we use branches, directories, or different repositories to store the desired state of different environments?" We'll explore the pros and cons of each approach, focusing on the best practices for maintaining application and environment-specific manifests.
#GitOps #DevOpsPractices #ArgoCD #BranchingStrategies
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please visit devopstoolkit.... for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopspara...
💬 Live streams: / devopsparadox
I love your chain of thought - I had to fight uphill battles in multiple jobs to *not* use long-lived branches, especially not production branches different from main.
I actually have a meeting scheduled with colleagues next week to explain exactly this to them - many thanks for creating something that allows me to avoid a 'meeting that should have been a youtube video' 😀
I'd love a deeper dive in file organization across organization. Kubernetes config, common CRDs, templates, common pipeline steps, etc. Would love to see a vertical slice of full enterprise repo/folder/file structure from the ground up.
In some projects, we typically use combination of both, we use a seperate overlay for each of the environments to define manifests, but we also use different branches just to have a promotion mechanism between the envs, so these branches should be eventually in sync.
We came to this setup after trying both ways, and the branching strategy is only to have a proper promotion mechanism and no other reason.
Are you kidding or can you read my search history. I just started today a look up for the topic of the video.
Therefore I am very interested and excited to watch your video
Great video again!
Here we use a repository per application, with everything (monorepo...), like backend services code, frontend code, helm charts, etc. We then have a separate repository with helmfile (very great tool) and a directory per environment. As the same team is responsible for test, staging and production environments, no need for separate repos!
I don't think that you should keep the charts in a separate repository. Put it where the apps are. If you're using a monorepo, put it there, just as you (probably) keep your build scripts, pipelines, tests, and other stuff in there.
Helmfile is not a good choice if you're using one of the GitOps tools since they don't need it and, in a way, replace it (and more). You can just put Argo CD or Flux manifests instead of the Helmfile (in that same repo). Those manifests should reference your charts and overwrite whichever values are specific to whichever environment you're syncing it to.
@@DevOpsToolkit The charts are indeed in our monorepo. It's been a long time that I want to try ArgoCD, will have a serious look at it soon! Thank you very much for taking the time to answer, I really appreciate it!!!!
We use 3 different approaches. Dir for cluster specific services. Dir for internal apps. Files for common services. And finally, a gitsha to point to last known good config of the common services for controlling when production env gets updated
Question Thursday:
How many GitOps (ArgoCd) Instances recommended when you staging about multiple environments and different Kubernetes versions?
Further details on the scenario: If you have for example 5 environments (Dev, function-test, QA, PreProd, Prod) and around 400 Apps Management by 10 Teams. A major/minor update of the Kubernetes version is not done via in-place update, but via a parallel Kubernetes instance next to the environment and the apps are moved. Is one ArgoCD instance enough? Should you separate Prod and Non-Prod? Should you separate Kubernetes updates in a new ArgoCD? Should you separate GitOps/ArgoCD instances per team or per namespace?
Is there antother recommendation if you want to use Argo Workflow or Argo Rollouts.
Is it than a good idea to split the Argo Instance between environments
Thanks
Great questions. I'll add them to one of the upcoming videos.
I would love a (visual) example of each way in action to have a better understanding
I'll do my best to make a similar video with visuals.
I once worked for a company which used branches for different environments. Contained Terraform & Manifests. They ended up with multiple "prod" branches somehow.
Nice series!
It would be amazing to se your preferred way to do gotops. With examples.
Thanks!
Can you give me a bit more info? Are you looking for a GitOps quickstart or something more concrete?
Love you and your Channel
💯 Separate folders for separate environments (/application sets). Dedicated branches per environment makes me wanna off myself.
Totally agree!
you can use b ranches for different environment manifests that are mergable - having an environment specific named manifest can exist in all branches but can be targeted in cicd when a commit to the required branch is detected. this would be pretty much the same as the environment specific directories you mention.
in this way you can have all manifests mergeable to the main branch and that wont interfere with each other.
That's true but, in that case, I'm not sure why use branches if targeting those same files in the mainline produces the same results (which is what I do). Now, to be clear, I'm referring to repos that contain only GitOps manifests and, in that case, workflows are not triggered by commits since those commits result in Argo CD synchronization which might not start, and certainly will not end, at the time commits are made. So, if you're referring to workflow runs that validate (test) deployments to, let's say, production, those must be triggered when the app is rolled out (unless you have tests that want to validate the state before, during, and after a rollout of a new release).
@@DevOpsToolkit it doesnt have to produce the same results - presumably you have branches to test code changes. targeting those files based on branch and thus code is the point. the different environment specific manifests can be used for more than one branch - eg i have used the dev manifest for all non named environments - that way devs can test their changes in a known environment but dont have to merge to the dev branch. the dev branch is just the first step in code promotion - eg dev > test > staging > uat > prod might be the code promotion path. this would inbclude both app and infra code.
One other supplementing rule is that ANYTHING that is deployed ANYWHERE should first touch mainbranch. If it's good enough for ev testring it's good enough for main branch
I have recently been struggling to work this out for myself, so thanks for the great info! The argocd best practices say "Using a separate Git repository to hold your Kubernetes manifests, keeping the config separate from your application source code, is highly recommended". It seems simpler to keep the application manifests together with the source code and just keep the environment configs in a seperate git repo. Any idea why they so strongly suggest the separation and have you hit any big problems by not doing it? As a gitops noob (but experienced dev), it's really difficult to filter all the online info to build a good dev-to-prod gitops pipeline.
For ne it Is very simple. I have to be able to write code, build it, test it, and run it. To do those things without going crazy, i have to have code, build scripts, tests, abd manifests in the same place (same repo). If manifests are going somewhere else, it would be logical to move the rest as well and that would be just silly.
My question: how do you deal with devbox (global) paths in the nushell? eval "$(devbox global shellenv)" does not work for nushell.
I love your videos, thank you for the time and effort!
I don't use devbox globally. I have a few tools I need globally (e.g. git) installed with brew and then everything else in repos.
Dear Viktor, please show us how to use Kubebuilder, how to create own CRDs for Kubernetes.
I'm not sure I can show it in a quick video like this one. I'll add it to my TODO list for a long one (those I publish on Mondays).
@@DevOpsToolkit I really appreciate that 🤝
I can't promise the exact date. I expect it to go live in 4-6 weeks.
Why can't we use different branches for different environments while keeping configurations/manifests for different environments in separate folders? One does not necessarily exclude the other. Each environment will 'know' from which folder it should be provisioned, while CI triggers are configured to watch for changes in specific branches. This will make all branches mergeable with one another and make propagation of changes quite trivial and git-driven.
I mean, it doesn't necessarily have to be configured like that, but it's definitely a viable option.
That is, more or less, the same as if you keep it in different directories in the mainline, so I'm not sure why would there be a branch, unless you're talking about creating a short-lived branch with changes to any of those so that they can be validated on a PR before they're merged to the mainline.
@@DevOpsToolkit I'm talking about CI/CD that is configured to deploy changes automatically to the corresponding environment as soon as they are pushed to one of dev/qa/stg branches. This assumes a CI/CD setup in which you don't need extra tools to propagate code changes to different environments: git merge and push are enough.
But here are some issues, if you are using Gitlab. That CI workflow will always trigger so lets say you make a new release artifact push then as a result of the new artifact you update your gitops manifest tag. That means the CI workflow will trigger again, though we dont want that because the source did not change?
I'm advocating for gitops manifests to be in a separate repo referencing base manifests in app repos. When you change a release tag you're changing it in the gitops repo and that change is likely going to be triggered by a webhook from the app repo.
As an unrelated question from someone who does not use GitLab, isn't there something equivalent to `[skip ci]` in GitHub?
@@DevOpsToolkitFrom GH docs:
Workflows that would otherwise be triggered using on: push or on: pull_request won't be triggered if you add any of the following strings to the commit message in a push, or the HEAD commit of a pull request:
[skip ci]
[ci skip]
[no ci]
[skip actions]
[actions skip]
That's what I was referencing. I was curious whether something like that exists in GitLab since @gardnerjens mentioned issues with that workflows would "trigger again".
When manifest on the same repo as source, I configure the trigger of the ci workflow to ignore change on some path (manifest, chart's value files, ...).
On github it's something like
```
on:
push:
branches:
- "**"
paths-ignore:
- chart/**
```
How do you feel about Kargo using branch-per-"rendered manifests" as an option? Their intent I think is to keep all environment configuration in mainline but render the final into an env-specific branch for sync only, i.e. it's never merged back to mainline.
I think thing that's a good idea. Also, I never grew to like Kargo. It does not feel right (and I'm probably wrong on that one).
@@DevOpsToolkit I'm yet to try kargo out properly (if ever) but would be keen for you to elaborate on what doesn't feel right in a vid :-) .
Not sure if those feelings relate to promotion but I personally opt for using Azure pipelines to manage CD promotion of CI manifest artifacts using PRs, generating the final manifests for review. Argocd watches the final manifests folder in mainline only, but could just as easily merge to an env-branch folder to keep the mainline clean of final manifests.
I'll do my best to create a video that goes into it in more detail. Until then, what you mentioned is one of the reasons I never developed "warm" feelings for it. I have to run workflows to build, test, etc. It's almost no effort for me to clone code of a repo, execute `yq` to change a tag, and the changes back to the repo. If I would not be using workflows for other tasks, Kargo would make much more sense but, since I do, one of the main advantages of it (update manifests) disappears. At the same time, I don't have to change the way I organize my manifests to comply to whichever organization Kargo expects me to have.
@@DevOpsToolkit Yeah it's all the other (smoke) workflow steps that often need doing post-deployment that I wasn't sure if kargo could integrate simply. And like you said, if that has to be done elsewhere anyway, then changing tags (or whatever for a promotion) is the least of the problem. Of course, determining exactly when those post-deploy steps can occur is its own problem 🙂
Thanks for the input. Looking forward to your next vid.
Can I avoid the massive amount of code duplication i.e directories containing basically the same thing?
Yes you can. Store manifests in one place and have the argo CD app overwrite values that differ from one env to another. That way you have one copy of all the manifests and have to duplicate only the argo CD manifest with each of those copies mostly unique since the big part of them are those unique values.
I don't know if I trust the devs with app manifests in their own repo....
That's what testers were saying in the past, yet most of us now have tests in app repos. The same can be said for workflows, build scripts, etc.
If you don't trust your fellow colleagues, there might be a problem that goes beyond GitOps or any other tech.
What would be a viable strategy to enforce promotion through environments? I'm tired of seeing the same modification to both, staging and production environment, in the same PR :(
If staging and production are always the same, I'd keep it all in the same place and point Argo CD (or Flux) in each of those envs. to the same location. I'm guessing that's not the case since there would not be much use to have staging that is the same as production.
As for the promotion... Change the tag in the staging manifest, test, change the tag in the prod. environment, test again.
A note: best practice (as even written in ISO standard) is to always have clear separation of production from other environments. That said that separation doesn't need to be a separate file, but as Victor mentioned can be achieved by tags, artifact deployments and a variety of other ways
We solve this by using kustomizations with a 'common' base layer and then overlays for staging/production etc. But I've also seen it done in a more basic way with symlinks inside git repos.
Thank you so much for this! Exactly what I have been trying to figure out. I really want to keep the application manifest configuration in the same repo as my app code. I plan to use KCL to create the manifests and abstract away k8 resources while setting good defaults and common config. One thing I am confused on is the deployment process with the gitops repo for my application. Should I publish my app KCL (kinda like helm chart) as a semantic versioned package then render that package in my gitops repo using environment values stored there as input into KCL options? Basically like creating a chart in apps and the values.yaml is in gitops repo, and you render that chart to a folder in the gitops repo instead of directly deploying helm to argo. I am afraid of using argo apps that point directly at the head of the app repo. If I make changes to the KCL code that is not ready for production, I wouldn't want argo to think that it can deploy those changes to prod.
I use kcl a lot but only to simplify writing manifests. My workflows are outputting kcl to yaml and storing them in git. An important note, in my case, is that most of my definitions are in clusters (CRDs abd controllers) so what's in repos are mostly simple yaml files (generated through kcl). They rarely exceed 20 lines in total.
@@DevOpsToolkit that's good to know. I want to make a simple interface for developers to create apps without writing yaml since they are used to writing terraform and deploying lambdas. I am trying to create a similar experience for k8s apps and allow them to configure env vars, ports, secrets etc without knowing about services or deployments. There could be several apps communicating with each other all stored in the same repo. I could make this abstraction through kcl or crossplane compositions. Either way, if a dev decided to add a volume mount to an app, how would we ensure that the change only deploys to lower environments and not prod before it should?
@Mvvement if i understood it correctly, you have some form of abstraction (e.g. CRD) that allows volume as input but that should be allowed only in specific environments. So, thr controller itself does not know when that input is acceptable and when it's not. If that's the case, than the best bet is to have admission controller policies that allow such input in some and fissalow it in other environments.
I'll do the best to dive deeper on that subject in one of the upcoming videos like the one you commented on (shorter abd focused on a single question).
@@DevOpsToolkit not exactly. The volume is allowed in all environments, I just want to ensure the change is promoted when expected. Say there is an app that is already deployed to all environments, dev, stg and prd. A developer creates a new feature for the app and adds this volume or any other change to the manifest to support the feature. Let's say this manifest change is only compatible with the new image. Once they commit the changes, how do I ensure the infrastructure in upper environments doesn't sync those changes from the base?
I'm not sure I understood so the answer but be silly. Correct me if I'm wrong...
You'll create s PR of the changes to the code, including the changes to the manifests (if needed). That PR will build and push a new image, deploy it to an ephemeral environments, abd test everything. If the change is valid, if it passed all the tests, it will be merged to the mainline and deployed to production (unless there are some other intermediary permanent environments). If that PR contained changed to the manifests, they will be applied to production as well (but after they were validated in an ephemeral environment). Whether that includes volumes or anything else is not important since you know that it will work since you tested it.
I'm sure I misunderstood so please correct me.
Hard disagree on the directories per environment or separate repository. Git is smarter than that, you can perfectly make changes to an environment-specific branch and still merge in new changes from lower environments without them affecting or overwriting the other environment. Directories per environment always ends the same, it's the approach I've seen the most at clients, and it always ends up in a horrible configuration drift between environments, which then it becomes the platform team's problem for some reason.
Using git branches has many advantages, and there is not a single reason why you "need" to merge to a mainline branch in Git. It for example allows you to implement specific branch permissions, which for some environments can be a requirement, or you could only allow a bot and a platform/operations team to update versions from a lower env to specific branches in acceptation or production environments, for example a bot that checks if an integration test suite was successfully run.
Say you have a development, test and production environment, you start off in your dev branch, with your base application manifests and specific configuration, from there you create a tst branch, where you make the necessary changes to the configuration, from there you branch off to prd to deploy to production and do the same. Any time you then merge changes from dev to tst or from tst to production, will preserve the branch-local changes. If there would be configuration conflicts or env-specific config changes when doing that from dev to tst for example, you branch from your tst environment branch, merge in the lower environment branch, make your changes, and merge that new temporary branch into your tst branch. This ensure your configuration is as close as possible between environments and you don't end up with "but it worked in the dev environment" - which has become the new "it works on my machine".
Is it perfect? No, but when having to deal with dozens of development teams deploying hundreds of services, being able to enforce a way of working prevents it from becoming a massive burden on the ops/platform team because dev teams screwed up.
Assuming that you specify in Argo CD Application manifest environment-specific variables (e.g., host), directories will not be the same. Based on your comment, I'm guessing that's not the case so I'm curious where you set environment-specific variables.
Also, based on your comment, I'm not sure I understood where you keep the base manifests. Are they in the app repos or in a "GitOps" repo? If it's the latter, that is the fundamental difference between what you're doing and me.
@@DevOpsToolkit I prefer the approach that for every single application/service having there are 2 repositories, one source code repository, and one "gitops" repository with branches per environments. In small environments you could probably have a single git repo containing all service configuration, but that doesn't really scale well, and blocks you from doing fancy stuff like auto creating Argo Applications in a certain env when a branch with a certain name is created, and routing traffic there using a service mesh for client requests with a certain header set.
Not sure what you mean with "base manifests", but most of the time, my approach is that teams use a base helm chart they include as a dependency in their gitops repo, and configure that using the values file and optionally some additional templates and files for specific customization. These base helm charts are versioned and pushed to a helm repository.
And naturally there are differences between the branches for each environment, but once changed for a specific environment/branch, it will not be overwritten when merging in new changes from a lower environment to that branch that don't touch those specific lines of configuration, and with the proper branch protections and pipelines, you can't have the "oops we forgot to add this new config part to production", or not updating this base helm chart version with changes they rely on. And on top of that, so many things here can be automated.
One of my big gripes with the whole gitops world is that it's called GITops, not copy-paste-ops. By using Git, you get a wealth of tools that I feel most of the gitops-y setups are completely ignoring because git is often not well understood.
@koffiezet By base manifests I mean manifests for an application (yaml or helm or customize or any other format with all the resources like a deployment, service, ingress, etc.). The only thing missing there are variables or overwrites with "stuff" specific to an environment (e.g. host, tag, etc.). I keep those in a separate environment specific repo and those are Argo CD Application manifests that not only overwrite but also point to wherever the "real" or base manifests are.