you also may want to use stakater/reloader to rollout restart your apps so they start using the new secret also, keep an eye to your cloud secrets manager bill as the constant monitor will start adding costs
@@DevOpsToolkit It depends on the situation, in my curret status (migrate old apps), our first step was to put config in env vars and with Banzai Injector is pretty easy to put in vault an env var inside a gitops repo with this kind of content: ENVAR: "vault:mystorage/myteam/myitem#mykey", you can aso create secrets with another syntax ENVAR: ">>vault:mystorate/myteam/myitemykey". And can also inject iit in the middle of an envar or configmap. But External Secret Operator did not exist when I began migration so at the moment I could not evaluate what's the best solution. For sure I will try External Secret Operator when I have time.
"If you have a problem, if no one else can help, and if you can find them, maybe you can hire the A-Team." and now you have got me the title theme of the A-Team stuck in my head! 😉🤔
I definitely love our external-secrets integration with Hashicorp Vault. It makes management much easier, as long as you have a single source-of-truth. It takes some getting used to, but always create/manage secrets in the external store and sync them using external-secrets. Next up: creating an operator to restart pods when a DB secret changes!
@mtik000 Using Hashicorp Vault Injector the secrets can be updated dynamically without requiring the pod to be restarted. It monitors the Vault secret for changes and injects the new one into a specific path in the pod (templates are also supported). Works great for me. I have also tested Secret CSI and worked in the same as Vault Injector (injecting the secret into the POD), but give also the option to create secrets resources inside K8S. But I missed the template option which is a very handy feature (maybe it does and I overlooked).
@@hutgerhauer The part I'm talking about, though, is about getting the application running inside the pod to use the new secret. Our apps generally only load the secrets on startup. I prefer not to have the secret placed anywhere on the pod itself (or in a k8s secret). The applications themselves authenticate with the Vault server using the `jwt` mounted to the pod, then read whatever secrets they need directly from Vault. I think it either scenario (yours or mine), the application needs to understand the secret has changed.
@@mtik000 got you. Well, if the application is not able to pick the newly injected secret without restarting, there's no point on using solutions such as Vault Injector or Secrets CSI. In my case, the apps do... they are able to detect a config file change and reload the settings.
At this moment I want to bring to your attention Doppler. (Not affiliated anyhow) It is a Secret Store (SaaS) which as a service is pretty good and feature-complete. It has a great web UI, straightforward CLI, and integration with many services. Now, Doppler has its own Kubernetes operator that does what external secrets do. You create one k8s secret with access token to Doppler and then a DopplerSecret which creates a k8s Secret. When I make a change to a secret in Doppler, it propagates immediately. On top, it has support for restarting deployments that are referencing secrets that were modified by adding one annotation. No need for Stakater/Reloader. That being said, there’s are 3 cons I find with external secrets, compared to Sealed Secrets: 1. GitOps framework is “violated” by not being able to follow secrets stored in External Password Stores. That means there will be 2 sources of truth (git repo + Secret Store, etc) that need to be compared in conjunction to understand when some change happens to the cluster. 2. In k8s manifests files, the k8s-native secret is not created as a resource (by us). I found the lack of support of secrets in k8s tools (like the k8s plugin in Intellij) annoying, since it will not recognize the secret that the Custom Secret resource is creating. It will mark it with a squiggly line. 3. Finally, you have no way to know what kind of secrets (which keys) are stored in an external secret, without looking at it inside the cluster. Like in your video, where you have username, host, password, port. When writing your manifests, you will have to go either in your Secret Store or use kubectl to find out what exists. Long text!! Sorry! 😅
Thanks for the recommendation. I'll check out Doppler ASAP. 1. You are right on this one, but many other resources are violating that. A reference to an image in, let's say, a Deployment. Even sealed secrets are violating that (in a way). The secret itself is not stored in Git but generated inside the cluster (by encrypting the value from the SealedSecret manifest). To be clear, I agree with you and, personally, I prefer SealedSecrets. They are more GitOps-friendly, but they also come with their own set of issues. 2. I think that's the nature of Kubernetes. We tend to create higher-level resources that a controlling the "real" ones. We create a Deployment that creates ReplicaSets, that create Pods, that manage containers. Similarly, Ingress is managing external load balancers while Crossplane or ClusterAPI are managing all sorts of external resources. We rarely manage "stuff" directly but through controllers that are in charge of managing the actual resources (often indirectly). 3. I think that SealedSecrets are not much different. Both are creating "other" resources that eventually create Kubernetes secrets. The major difference is that one (SealedSecrets) is decrypting secrets inside the cluster while the other (External Secrets) is pulling them from somewhere else (just as containers are pulling container images). In both cases, we cannot know what the secret is by inspecting the manifest. Now, if you're referring to individual fields of a secret, they can be specified with External Secrets as well. I just used a shorter version saying "give me everything because I stored everything as one big secret in GCP".
The 'I'm not touching my cluster' frame is meme-worthy. Just saying.. ;) Anyhow, akuity recently did a webinar on the subject of gitops-friendly secrets management, and the external secrets operator does seem to be the "best" option. Argocd has some plugins too.
I haven't tried it yet. In general, I do not think that projects like Argo should be concerned with secrets. We already have solutions that manage them well and I'm sceptical whether side projects like that can out as much attention to it as those fully focused on secrets like external secrets, sealed secrets, and others. Nevertheless, let me check it out before saying anything about it.
Hi Victor, would you recommend some tool to keep a track of running services / ports / IPs etc... on Kubernetes cluster... something like "live" documentation, if anything like that even exists... I know of Backstage from Spotify but the deployment of that is kind of strange. I'm looking for something I can deploy to Kubernetes, and people can go there, look up app running on the cluster, find the owner, some documentation to it and so on...
My answer will be biased... I tend to use crossplane which, through compositions, gives me different ways to groups resources and expose them through CLI/UI.
I'm now using external Secrets Operator because sealed secrets failed me big time :-( I upgraded k8s version in a cluster using sealed secrets, and after the upgrade I could not unseal the secrets anymore... I know, it should not happen but it did :-( I had to recreate all sealed secrets again (no i did not had a backup... but since the issue it seems was the k8s cluster certificate change not sure if it would have helped ). Has this happened to anyone ? (Hint: only seems an issue when using strict... at least with cluster-wide it never happened.)
If you have a multitenance cluster shared between different teams and you have a vault instance shared between the different teams, you would need some kind of permission management or control which team may read which endpoints from the vault. Otherwise everybody can read everything (unless that what you want?). I think that Secret store CSI driver is much more mature solution where you may define vault policy roles based on k8s namespaces and map those roles to the teams.
Hey B K! Vault Policy Roles with Secret Store CSI driver are great indeed. However, ESO supports this approach ever since version 0.1 (since the beginning ). You can create multiple SecretStores, each of them bound to a given role. I would like to invite you to please look for multitenancy setups on External Secrets Operator docs (I cant point you there since youtube will delete the comment).
Hey man, great video! Helped me understand ESO a lot better :) There is one thing I'm struggling to find out though, and I'm hoping you might have an answer for me: If I want to have keys managed by Azure Key Vault to auto-rotate every X days, and I use ESO to fetch these auto-rotating keys (with a refresh interval of 1h), then what would happen if - ESO refreshes the secrets from Azure Key Vault - Azure Key Vault rotates my key at scheduled time, which happens to be 1 minute after the refresh from ESO Will my applications that use this key now not work for 59 minutes? I've really been trying to figure this one out, but there seems to be absolutely no answer to this anywhere online. :(
When you rotate keys, you still need to keep the old key operational for a while. Otherwise, no solution would work. For example, let's say that you have authentication to a db and you rotate it. If that would mean that from there on only the new auth works, all currently established connections would fail.
I've been overlaying resources rather than values. It might be possible to overlay values as well, but I haven't tried that option, mostly because I believe that overlaying resources is closer to the "Kustomize spirit".
@@DevOpsToolkit thank you for the content and explanation Victor. Could you make a video for this specific scenario of "secrets rotation + keeping the old value" to avoid donwtime until the app gets the new value? And also when we have N replicas of the pod using that secret.
Victor, did you already consider a video on (cloud provider agnostic) secret stores? I would really love to see one from you since you are focusing on K8s and just do awesome content
I do not think there is anything better than ESO right now. The only alternative that i am aware of is SealedSecrets and SSCSID. SealedSecrets does not work with secret stores like vault and i am not sure that SSCSID direction is a good one for all use cases. If you know of any other solution, i would be very happy to give it a spin.
@@DevOpsToolkit thanks for the input! I really like ESO, it rather is Vault that gets on my nerves. If I find out about something else, I will let you know but I guess you will be the first one to know, anyway 😄
I just realized that you asked about secret stores and not secret managers. So, my previous answer must have been confusing. For secret stored, Vault and those provided by hyperscalers are the only ones i see being widely used and I haven't seen any challenger on the horizon 😔
The ESO docs for Hashicorp Vault mention that "Vault itself implements lots of different secret engines, as of now we only support the KV Secrets Engine." Did you manage to make it work with any other Secret Engine except KV? I find this pretty limiting as I have to implement dynamic secrets and secret rotation myself although Vault offers it for most if its engines.
Hi sir , I need a solution for the infra setup I'm using. I'm using external secrets operator for secrets management Consider I'm dynamically creating new envs for testing so that they are dynamically creating external secrets and kubernetes secrets for the service (frontend or backend)which will get provisioned . Here in my AWS secrets manager I have templating in the values of keys. Example:- db_name : tf-{{ ENV_NAME }}-{{ BASE_DOMAIN }} So while creating kubernetes secret from external secrets I want to dynamically replace this {{ ENV_NAME }} and {{ BASE_DOMAIN }} with the namespace that I'm dynamically creating for the new env that I'm provisioning and xyz respectively. How can I include templating logic in external secret file so that it will Directly create a templated kubernetes secret file .
The con taht you mentioned - how offten are you needing to push secrets form kubernetes? When does one create a secret from within kubernetes if kubernetes' secrets manager is not so great?
Let's say that you create a database on one kubernetes cluster (either inside that same cluster or in a hyperscaler). Further on, imagine that the app that should use that db is in a different cluster. In that scenario you might want to push that secret from the first cluster into a secret manager (where it is) and pull it into the second cluster.
@@DevOpsToolkit oh i see. is there a best practice when creating secrets from within kubernetes? from my experience they start off pretty insecure. haven't dealt with them much
@RockyScenes in most cases secrets are created by third-party apps. For example, if you create a DB with Crossplane it will create a secret with authentication to that db. The exceptions tend to be secrets like authentication required for third party tools (e.g. auth crossplane needs to speak to AWS). In those cases I tend to pull them with ESO from whichever secrets manager i use (unless I use IRSA). An alternative could be SealedSecrets that does not require secrets store but encrypts manifests allowing you to store them in git and decrypts them inside the cluster.
I was expecting something about Secret Store CSI Driver, but External Secrets Operator is also interesting. But what happens on the application side when the secret is updated?
I was watching another intro video about External Secrets and the presenter said that Sealed Secrets doesn't scale well, but he didn't point out why. I'm wondering why he said that (and rereading this comment, I might have answered my own question). The missing secrets generation and store to secret stores point you make is also why I feel Sealed Secrets are a "better fit" for what we want to accomplish. However, theoretically, we might have a large number of clusters, so that scaling comment does raise concerns. Correct me if I am wrong, but if the generation of the sealed secret and (controlled) storage of it in a git repo along with the generation of the values to be encrypted themselves are all automated (i.e. no manual sealed secret generation), then what is to stop the process from scaling out to 1000's of clusters, when each cluster has its own sealed secret controller? This is also assuming the apps using the sealed secrets are cluster specific too. I realize, for apps that might be used across large amounts of clusters, sealed secrets wouldn't scale well. Is that what he meant? For that though, I'd use a different pattern/ tool, maybe even ESO? Btw, I also looked at the ESO docs and it doesn't seem to have a method to store back to the external secret stores yet. How can the generation of secrets even be automated then? To me, that is a real scalability issue.
I'm not sure what he meant when saying that sealed secrets do not scale. The job is relatively easy and is performed only when manifests change to there is no constant demand for resources. Also, i haven't experienced any performance/scaling issues myself.
@@DevOpsToolkit Awesome. Appreciate the quick reply. From all the solutions out there, sealed secrets fits best for what we require. I am also now liking the idea more and more to store the app "target" state in the clients' own repos.
Hi Victor, thanks for another interesting video! How does the complexity of configuring external secrets fair vs bitnami sealed secrets? My team is transitioning to k8s, so we're looking for solutions that are easy to install / manage and would give us immediate benefit
In case of External Secrets, the complexity is in managing secrets in an external store (e.g., HashiCorp vault) as well as maintenance of the store itself (if it's not SaaS). If we ignore that, External Secrets are pretty easy and straightforward. SealedSecrets is easy as well and there's nothing to manage but secrets themselves stored in Git. Both are great and neither is overly complex. The main question is whether you prefer to keep secrets in an external store or in Git (an encrypted version of them).
The major difference is that it does not create kubernetes secrets but mounts volumes which should be more secure (kubernetes secrets are very insecure). On the other hand, that is not a common approach in kubernetes and might cause issues with third party tools.
Actually, crossplane is one of the reasons I explored external secrets. I need something to pull secrets into my clusters so that crossplane can use them but also something to push secrets generated by crossplane into my secret store. That's why I mentioned "secrets push" as the only missing feature. Once that is available, the secrets generated with crossplane will be available wherever I need them.
@@DevOpsToolkit I'm very curious about how you combined Crossplane with External-secrets. I generate kubernetes clusters with Crossplane and get the kubeconfig for them in crossplane-system namespace. I tried with Sealed Secrets but passing the private certificate to decrypt secrets around different clusters is a pain. The feature to push secrets is now available with External Secrets, so what would the approach be? Just using a kubernetes provider and push the secret from the management cluster into the created clusters?
@@IvanRizzante Getting secrets into k8s is easy. Pushing them from k8s to a secret store is a challenge. A while ago, I was urging ESO community to add "push" feature to the project and they did that. However, I did not yet have enough time to dive into it.
Thank you for all your video content. This is something I'm looking to implement on our cluster soon. I have one question though. When it comes to storing trustore.ts files as secrets, how would we do this with AWS Secrets Manager. Should we base64 encode it? Any help would be greatly appreciated.
All secrets in Kubernetes are base64 encoded, so yes. You would need to create that secret first either with kubectl or anything else. However, once the secret is there, you can manage it with external secrets.
This is great to showcase operator outreach however I am not still convinced what happens if the cluster is exposed by some chance as we are syncing secrets from external secret management services which essentially syncing the external secret object(copy) into Kuebernetes cluster and making it available as another dumb Kuebernetes secret anyone can decode with base64 -d. Sorry, in case I have misconstrued the video and topic. I am more interested how ESO handles encryption at rest not just the plain naked secrets stored in SM.
ESOs responsibility stops when secrets in kubernetes are created. From there on, it all depends on permissions you defined, whether etcd is encryped, etc. The alternative is bot to use kubernetes secrets at all. I will publish a video on that subject in 3-4 weeks. TL;DR; it is close to impossible not to use kubernetes secrets.
Can it wildcard the secret names and bring in all the secrets from the source. I don't want to specify them by name. I want to selectively override certain default values from the app config, but I want to leave out the secrets from my vault that I don't care about.
@@DevOpsToolkit Right, that's what I thought. What I'm focusing on right now is app configuration, which needs about 10 or so values, some of which are sensitive, some not. They all have good defaults, so if you don't give a value, the app uses the default. So what I'm thinking is I only need one secret in Kubernetes ... the connection string that gets them to Hashicorp Vault, Azure Key Vault, AWS Secrets Manager, etc. All other config settings go in the external vault. That way they can change config values after-the-fact and they don't need to keep going back to their Helm charts or K8s manifests. It would be a real hassle for the user to need to keep going back to k8s just to change one simple config value. So if I only need to store one secret, then external-secrets isn't that much value. Sounds like more trouble than it is worth.
thanks for the video. I'm using AWS and by default the secrets are encrypted and then the secret is not usable (trying to use it with Airflow GitSync side cart) Could you suggest a solution for my problem?
@@sharonlavie8210 Normally, values in k8s secrets are encoded, not encrypted. Can you confirm that's not the case? You can do it with something like `echo -n "VALUE_FROM_THE_K8S_SECRET" | base64 -d`.
@@DevOpsToolkit yes, you're right!! it is base64 encoded. just need to figure out how to decode the values i get from the secret when i use them in my gitsync sidecar. You are #1!! I really enjoy watching your videos and I learn a lot from you. Will increase my support in your channel, you totally deserve it ❤
I found the solution. very simple, the username and password to github were base64 encoded, i saved them without the base64 encoding in AWS Secrets manager. Problem solved!! yay! :)
That depends on whether you do use a secrets store or not. Unless you're a "small shop", using secrets store is the way to go and, in that case, ESO is, in my opinion, the best choice. Otherwise, Sealed Secrets is my choice for managing secrets without a secrets store.
@@DevOpsToolkit I personally like the simplicity of sealed secrets, since there are no external dependencies and access control is only at the cluster level (or even only at the GitHub repository level if the person generating the secrets has the public key in hands, they dont even need acces to the cluster, they just encrypt the secret and push it to git). The only thing I can think of as an extra bonus for ESO is the PushSecret thing, but in my case the Platform team provided me with a cluster with Crossplane installed and all the stuff I need to deploy my application so Crossplane and my apps are the same cluster (not sure if its a good or bad idea though).
That's indeed a "chicken and egg problem" and I do not have a good answer for that except to say that once a secret is created in a "typical" fashion (e.g., `kubectl create secret`) it can be managed by External Secrets from there on. It's a similar problem to "how to install Argo CD without Argo CD" or "how to create a cluster where Crossplane will be running".
There is always some prerequisite for any tool. We're all trying to reduce those initial dependencies but they are always there in some form or another.
Having this secret within the cluster as an opaque, is safe? I mean isn't it more safe to have these secrets only loaded in runtime by the application?
The idea, in general, is for the apps to focus on business logic and leave everything else to external processes. That applies just as much to networking, storage, and TLS as to secrets. The result is the same and the only real difference is where the logic is coded.
@@DevOpsToolkit thank you for your reply, so basically in terms of security, keep these secrets on cluster are safer than on the apps, with a plus of isolating complexity from the development. Cool
I haven't used it sufficiently (yet). I've been using GoDaddy solution for a long time (External Secrets is a continuation of that project) so I haven't had the need to dive into Secret Store CSI. Is it good? Should I switch to it?
@@DevOpsToolkit Secret Store CSI Driver is a k8s standard with lots of supported integrations (e.g., ASCP is Amazon's provider). In many cases you can avoid creating k8s secrets, as per best security practice, but creating a k8s secret is also supported.
@@joebowbeer I definitely need to go back to SSC. It must have changed a lot since the last time I checked it (over a year ago). Does it support pushing secrets to external stores? That's one of the main things missing in External Secrets (and I believe they're working on it).
Secret Store CSI Driver is great if you need to avoid having Kubernetes Native Secrets (or sensitive data in ETCD). This is specially useful with some specific compliances and Very specific sensitive workloads. Just need to remember that the service account token used by SSCSI driver is on ETCD anyways. We've seen people use both projects, even simultaneously, in the same organization. Remember: All your practices around access management, avoidance of initial attack vectors and having good identity boundaries will be way more important in avoiding people getting secrets than what tool you choose to synchronize secrets, or if they are in volumes or etcd :)
@@DevOpsToolkit They don't push secrets and I don't think it is in their roadmap, since their focus is being a Storage Interface. The focus of the SSCSID project is not to have a reconciliation loop creating/updating stuff. The provider (like ASCP mentioned here), is a privileged daemonset in the cluster, and SSCSID is just what the names tell us, a storage driver so kubernetes knows how to mount the secret as a volume in the pod. Which is why, even if you enable secret sync with SSCSID you would need to mount it first to a pod. So with image pull secrets that you usually would not use in a pod, you would have to first mount to a dummy pod so it ends up in a K8S native secret.
I think that if we believe 'each change should have a corresponding commit in Git,' then this method of secret management is not well-suited to the GitOps model. I prefer SOPS so I can handle changes to secrets like other resources. If the Git repository is the sole source of truth, then we should not allow other parties to inject changes directly into our infrastructure.
I do not think that git is the source of truth. It is the source of the desired state. The truth is the actual state which are resource at runtime. The actual state might or might not have drift when compared to the desired state. Also, Git often contains higher level abstractions. It might have a Deployment which, at runtime, managers replica sets which manage pods which contain containers. All those are the source of truth and not only the intention to have a Deployment.
As for ESO... It is not that different from, let's say, containers. While ESO references a secret which is later pulled into a cluster, a container definition references an image which is also pulled into a cluster and converted into actual containers.
How do you manage your secrets?
Banzai vault injector
@@oftheriverinthenight I haven't tried it yet. It seems similar to Secrets Store CSI (but limited to Vault). Do you recommend it?
you also may want to use stakater/reloader to rollout restart your apps so they start using the new secret
also, keep an eye to your cloud secrets manager bill as the constant monitor will start adding costs
ArgoCD + Argo Vault Plug-in to external Vault Cluster
@@DevOpsToolkit It depends on the situation, in my curret status (migrate old apps), our first step was to put config in env vars and with Banzai Injector is pretty easy to put in vault an env var inside a gitops repo with this kind of content: ENVAR: "vault:mystorage/myteam/myitem#mykey", you can aso create secrets with another syntax ENVAR: ">>vault:mystorate/myteam/myitemykey". And can also inject iit in the middle of an envar or configmap. But External Secret Operator did not exist when I began migration so at the moment I could not evaluate what's the best solution. For sure I will try External Secret Operator when I have time.
Great Video!
PushSecrets are being developed and will soon be available! Aiming for next release, or the one after that :)
Looking forward to it!!
"If you have a problem, if no one else can help, and if you can find them, maybe you can hire the A-Team." and now you have got me the title theme of the A-Team stuck in my head! 😉🤔
I definitely love our external-secrets integration with Hashicorp Vault. It makes management much easier, as long as you have a single source-of-truth. It takes some getting used to, but always create/manage secrets in the external store and sync them using external-secrets. Next up: creating an operator to restart pods when a DB secret changes!
Have a look over Reloader by stakater :)
@@knelasevero Interesting, thanks!
@mtik000 Using Hashicorp Vault Injector the secrets can be updated dynamically without requiring the pod to be restarted. It monitors the Vault secret for changes and injects the new one into a specific path in the pod (templates are also supported). Works great for me.
I have also tested Secret CSI and worked in the same as Vault Injector (injecting the secret into the POD), but give also the option to create secrets resources inside K8S. But I missed the template option which is a very handy feature (maybe it does and I overlooked).
@@hutgerhauer The part I'm talking about, though, is about getting the application running inside the pod to use the new secret. Our apps generally only load the secrets on startup. I prefer not to have the secret placed anywhere on the pod itself (or in a k8s secret). The applications themselves authenticate with the Vault server using the `jwt` mounted to the pod, then read whatever secrets they need directly from Vault. I think it either scenario (yours or mine), the application needs to understand the secret has changed.
@@mtik000 got you. Well, if the application is not able to pick the newly injected secret without restarting, there's no point on using solutions such as Vault Injector or Secrets CSI. In my case, the apps do... they are able to detect a config file change and reload the settings.
Your vids are the best ... I always learn a lot from your channel.. Thanks for sharing your knowledge :D
Very nice, I just deployed vault to my cluster, this came like on call :D
Great explanation Victor...Keep it up
At this moment I want to bring to your attention Doppler. (Not affiliated anyhow)
It is a Secret Store (SaaS) which as a service is pretty good and feature-complete. It has a great web UI, straightforward CLI, and integration with many services.
Now, Doppler has its own Kubernetes operator that does what external secrets do. You create one k8s secret with access token to Doppler and then a DopplerSecret which creates a k8s Secret. When I make a change to a secret in Doppler, it propagates immediately. On top, it has support for restarting deployments that are referencing secrets that were modified by adding one annotation. No need for Stakater/Reloader.
That being said, there’s are 3 cons I find with external secrets, compared to Sealed Secrets:
1. GitOps framework is “violated” by not being able to follow secrets stored in External Password Stores. That means there will be 2 sources of truth (git repo + Secret Store, etc) that need to be compared in conjunction to understand when some change happens to the cluster.
2. In k8s manifests files, the k8s-native secret is not created as a resource (by us). I found the lack of support of secrets in k8s tools (like the k8s plugin in Intellij) annoying, since it will not recognize the secret that the Custom Secret resource is creating. It will mark it with a squiggly line.
3. Finally, you have no way to know what kind of secrets (which keys) are stored in an external secret, without looking at it inside the cluster. Like in your video, where you have username, host, password, port. When writing your manifests, you will have to go either in your Secret Store or use kubectl to find out what exists.
Long text!! Sorry! 😅
Thanks for the recommendation. I'll check out Doppler ASAP.
1. You are right on this one, but many other resources are violating that. A reference to an image in, let's say, a Deployment. Even sealed secrets are violating that (in a way). The secret itself is not stored in Git but generated inside the cluster (by encrypting the value from the SealedSecret manifest). To be clear, I agree with you and, personally, I prefer SealedSecrets. They are more GitOps-friendly, but they also come with their own set of issues.
2. I think that's the nature of Kubernetes. We tend to create higher-level resources that a controlling the "real" ones. We create a Deployment that creates ReplicaSets, that create Pods, that manage containers. Similarly, Ingress is managing external load balancers while Crossplane or ClusterAPI are managing all sorts of external resources. We rarely manage "stuff" directly but through controllers that are in charge of managing the actual resources (often indirectly).
3. I think that SealedSecrets are not much different. Both are creating "other" resources that eventually create Kubernetes secrets. The major difference is that one (SealedSecrets) is decrypting secrets inside the cluster while the other (External Secrets) is pulling them from somewhere else (just as containers are pulling container images). In both cases, we cannot know what the secret is by inspecting the manifest. Now, if you're referring to individual fields of a secret, they can be specified with External Secrets as well. I just used a shorter version saying "give me everything because I stored everything as one big secret in GCP".
I love to use Doppler to store my secrets and use Doppler CLI on pipelines for manage and use them :)
You save my project, thank you so much!
The 'I'm not touching my cluster' frame is meme-worthy. Just saying.. ;)
Anyhow, akuity recently did a webinar on the subject of gitops-friendly secrets management, and the external secrets operator does seem to be the "best" option.
Argocd has some plugins too.
Hi Victor! Amazing video. Thanks! Just one question: what do you think about argo vault plugin? Is good? Or is it better to use external secrets?
I haven't tried it yet. In general, I do not think that projects like Argo should be concerned with secrets. We already have solutions that manage them well and I'm sceptical whether side projects like that can out as much attention to it as those fully focused on secrets like external secrets, sealed secrets, and others. Nevertheless, let me check it out before saying anything about it.
Hi Victor, would you recommend some tool to keep a track of running services / ports / IPs etc... on Kubernetes cluster... something like "live" documentation, if anything like that even exists... I know of Backstage from Spotify but the deployment of that is kind of strange. I'm looking for something I can deploy to Kubernetes, and people can go there, look up app running on the cluster, find the owner, some documentation to it and so on...
I like using Rancher to get an overview and use labels as well as annotations (plus Rancher projects that group namespaces) for organization.
My answer will be biased... I tend to use crossplane which, through compositions, gives me different ways to groups resources and expose them through CLI/UI.
I'm now using external Secrets Operator because sealed secrets failed me big time :-( I upgraded k8s version in a cluster using sealed secrets, and after the upgrade I could not unseal the secrets anymore... I know, it should not happen but it did :-( I had to recreate all sealed secrets again (no i did not had a backup... but since the issue it seems was the k8s cluster certificate change not sure if it would have helped ). Has this happened to anyone ? (Hint: only seems an issue when using strict... at least with cluster-wide it never happened.)
Thanks, that was helpful!
If you have a multitenance cluster shared between different teams and you have a vault instance shared between the different teams, you would need some kind of permission management or control which team may read which endpoints from the vault. Otherwise everybody can read everything (unless that what you want?). I think that Secret store CSI driver is much more mature solution where you may define vault policy roles based on k8s namespaces and map those roles to the teams.
Hey B K! Vault Policy Roles with Secret Store CSI driver are great indeed. However, ESO supports this approach ever since version 0.1 (since the beginning ). You can create multiple SecretStores, each of them bound to a given role. I would like to invite you to please look for multitenancy setups on External Secrets Operator docs (I cant point you there since youtube will delete the comment).
Hey man, great video! Helped me understand ESO a lot better :) There is one thing I'm struggling to find out though, and I'm hoping you might have an answer for me:
If I want to have keys managed by Azure Key Vault to auto-rotate every X days, and I use ESO to fetch these auto-rotating keys (with a refresh interval of 1h), then what would happen if
- ESO refreshes the secrets from Azure Key Vault
- Azure Key Vault rotates my key at scheduled time, which happens to be 1 minute after the refresh from ESO
Will my applications that use this key now not work for 59 minutes?
I've really been trying to figure this one out, but there seems to be absolutely no answer to this anywhere online. :(
When you rotate keys, you still need to keep the old key operational for a while. Otherwise, no solution would work. For example, let's say that you have authentication to a db and you rotate it. If that would mean that from there on only the new auth works, all currently established connections would fail.
I've been overlaying resources rather than values. It might be possible to overlay values as well, but I haven't tried that option, mostly because I believe that overlaying resources is closer to the "Kustomize spirit".
@@DevOpsToolkit thank you for the content and explanation Victor. Could you make a video for this specific scenario of "secrets rotation + keeping the old value" to avoid donwtime until the app gets the new value? And also when we have N replicas of the pod using that secret.
@thiagoscodeler5152 adding it to my to-do list...
Victor, did you already consider a video on (cloud provider agnostic) secret stores? I would really love to see one from you since you are focusing on K8s and just do awesome content
I do not think there is anything better than ESO right now. The only alternative that i am aware of is SealedSecrets and SSCSID. SealedSecrets does not work with secret stores like vault and i am not sure that SSCSID direction is a good one for all use cases. If you know of any other solution, i would be very happy to give it a spin.
@@DevOpsToolkit thanks for the input! I really like ESO, it rather is Vault that gets on my nerves.
If I find out about something else, I will let you know but I guess you will be the first one to know, anyway 😄
I just realized that you asked about secret stores and not secret managers. So, my previous answer must have been confusing.
For secret stored, Vault and those provided by hyperscalers are the only ones i see being widely used and I haven't seen any challenger on the horizon 😔
The ESO docs for Hashicorp Vault mention that "Vault itself implements lots of different secret engines, as of now we only support the KV Secrets Engine." Did you manage to make it work with any other Secret Engine except KV? I find this pretty limiting as I have to implement dynamic secrets and secret rotation myself although Vault offers it for most if its engines.
I used it mostly with Google servers and a bit with aws secrets manager so i don't hwveuch experience with it combined with vault 😔
Thanks!
Thanks a ton
Hi sir , I need a solution for the infra setup I'm using.
I'm using external secrets operator for secrets management
Consider I'm dynamically creating new envs for testing so that they are dynamically creating external secrets and kubernetes secrets for the service (frontend or backend)which will get provisioned .
Here in my AWS secrets manager I have templating in the values of keys.
Example:- db_name : tf-{{ ENV_NAME }}-{{ BASE_DOMAIN }}
So while creating kubernetes secret from external secrets I want to dynamically replace this {{ ENV_NAME }} and {{ BASE_DOMAIN }} with the namespace that I'm dynamically creating for the new env that I'm provisioning and xyz respectively.
How can I include templating logic in external secret file so that it will Directly create a templated kubernetes secret file .
Check out ESO templates. They should do the trick.
The con taht you mentioned - how offten are you needing to push secrets form kubernetes? When does one create a secret from within kubernetes if kubernetes' secrets manager is not so great?
Let's say that you create a database on one kubernetes cluster (either inside that same cluster or in a hyperscaler). Further on, imagine that the app that should use that db is in a different cluster. In that scenario you might want to push that secret from the first cluster into a secret manager (where it is) and pull it into the second cluster.
@@DevOpsToolkit oh i see. is there a best practice when creating secrets from within kubernetes? from my experience they start off pretty insecure. haven't dealt with them much
@RockyScenes in most cases secrets are created by third-party apps. For example, if you create a DB with Crossplane it will create a secret with authentication to that db. The exceptions tend to be secrets like authentication required for third party tools (e.g. auth crossplane needs to speak to AWS). In those cases I tend to pull them with ESO from whichever secrets manager i use (unless I use IRSA). An alternative could be SealedSecrets that does not require secrets store but encrypts manifests allowing you to store them in git and decrypts them inside the cluster.
@@DevOpsToolkit Rotating secret like AWS ECR could be another use case?
@@arwema true
I was expecting something about Secret Store CSI Driver, but External Secrets Operator is also interesting. But what happens on the application side when the secret is updated?
I already have as csi on my TODO list... :)
I was watching another intro video about External Secrets and the presenter said that Sealed Secrets doesn't scale well, but he didn't point out why. I'm wondering why he said that (and rereading this comment, I might have answered my own question).
The missing secrets generation and store to secret stores point you make is also why I feel Sealed Secrets are a "better fit" for what we want to accomplish. However, theoretically, we might have a large number of clusters, so that scaling comment does raise concerns.
Correct me if I am wrong, but if the generation of the sealed secret and (controlled) storage of it in a git repo along with the generation of the values to be encrypted themselves are all automated (i.e. no manual sealed secret generation), then what is to stop the process from scaling out to 1000's of clusters, when each cluster has its own sealed secret controller? This is also assuming the apps using the sealed secrets are cluster specific too. I realize, for apps that might be used across large amounts of clusters, sealed secrets wouldn't scale well. Is that what he meant? For that though, I'd use a different pattern/ tool, maybe even ESO?
Btw, I also looked at the ESO docs and it doesn't seem to have a method to store back to the external secret stores yet. How can the generation of secrets even be automated then? To me, that is a real scalability issue.
I'm not sure what he meant when saying that sealed secrets do not scale. The job is relatively easy and is performed only when manifests change to there is no constant demand for resources. Also, i haven't experienced any performance/scaling issues myself.
@@DevOpsToolkit Awesome. Appreciate the quick reply. From all the solutions out there, sealed secrets fits best for what we require. I am also now liking the idea more and more to store the app "target" state in the clients' own repos.
thanks, surely useful for companies with centralized credentials.
Hi Victor, thanks for another interesting video! How does the complexity of configuring external secrets fair vs bitnami sealed secrets? My team is transitioning to k8s, so we're looking for solutions that are easy to install / manage and would give us immediate benefit
In case of External Secrets, the complexity is in managing secrets in an external store (e.g., HashiCorp vault) as well as maintenance of the store itself (if it's not SaaS). If we ignore that, External Secrets are pretty easy and straightforward. SealedSecrets is easy as well and there's nothing to manage but secrets themselves stored in Git.
Both are great and neither is overly complex. The main question is whether you prefer to keep secrets in an external store or in Git (an encrypted version of them).
@@DevOpsToolkit awesome, thank you for explaining
Any idea how this compares to the CSI Secrets Driver? It looks almost the same.
The major difference is that it does not create kubernetes secrets but mounts volumes which should be more secure (kubernetes secrets are very insecure). On the other hand, that is not a common approach in kubernetes and might cause issues with third party tools.
Can you do a review on doppler? Both with the k8s operator and the cli approach (see their docs)
Adding it to my TODO list... :)
As a side note, I think we had someone from Doppler on www.devopsparadox.com
Thank you very much for the explanation. It seems to be a reverse crossplane for secrets 😅..
Actually, crossplane is one of the reasons I explored external secrets. I need something to pull secrets into my clusters so that crossplane can use them but also something to push secrets generated by crossplane into my secret store. That's why I mentioned "secrets push" as the only missing feature. Once that is available, the secrets generated with crossplane will be available wherever I need them.
@@DevOpsToolkit I'm very curious about how you combined Crossplane with External-secrets.
I generate kubernetes clusters with Crossplane and get the kubeconfig for them in crossplane-system namespace.
I tried with Sealed Secrets but passing the private certificate to decrypt secrets around different clusters is a pain.
The feature to push secrets is now available with External Secrets, so what would the approach be?
Just using a kubernetes provider and push the secret from the management cluster into the created clusters?
@@IvanRizzante Getting secrets into k8s is easy. Pushing them from k8s to a secret store is a challenge. A while ago, I was urging ESO community to add "push" feature to the project and they did that. However, I did not yet have enough time to dive into it.
Thank you for all your video content. This is something I'm looking to implement on our cluster soon.
I have one question though. When it comes to storing trustore.ts files as secrets, how would we do this with AWS Secrets Manager. Should we base64 encode it? Any help would be greatly appreciated.
All secrets in Kubernetes are base64 encoded, so yes. You would need to create that secret first either with kubectl or anything else. However, once the secret is there, you can manage it with external secrets.
Ok. thank you for your help.
This is great to showcase operator outreach however I am not still convinced what happens if the cluster is exposed by some chance as we are syncing secrets from external secret management services which essentially syncing the external secret object(copy) into Kuebernetes cluster and making it available as another dumb Kuebernetes secret anyone can decode with base64 -d. Sorry, in case I have misconstrued the video and topic. I am more interested how ESO handles encryption at rest not just the plain naked secrets stored in SM.
ESOs responsibility stops when secrets in kubernetes are created. From there on, it all depends on permissions you defined, whether etcd is encryped, etc. The alternative is bot to use kubernetes secrets at all. I will publish a video on that subject in 3-4 weeks. TL;DR; it is close to impossible not to use kubernetes secrets.
I still mess up things with Vault :)), I hope I will master it in one day
Can it wildcard the secret names and bring in all the secrets from the source. I don't want to specify them by name. I want to selectively override certain default values from the app config, but I want to leave out the secrets from my vault that I don't care about.
As far as I know, it cannot use wildcards. You need to be specific which secrets you want.
@@DevOpsToolkit Right, that's what I thought. What I'm focusing on right now is app configuration, which needs about 10 or so values, some of which are sensitive, some not. They all have good defaults, so if you don't give a value, the app uses the default. So what I'm thinking is I only need one secret in Kubernetes ... the connection string that gets them to Hashicorp Vault, Azure Key Vault, AWS Secrets Manager, etc. All other config settings go in the external vault. That way they can change config values after-the-fact and they don't need to keep going back to their Helm charts or K8s manifests. It would be a real hassle for the user to need to keep going back to k8s just to change one simple config value.
So if I only need to store one secret, then external-secrets isn't that much value. Sounds like more trouble than it is worth.
thanks for the video.
I'm using AWS and by default the secrets are encrypted and then the secret is not usable (trying to use it with Airflow GitSync side cart)
Could you suggest a solution for my problem?
Do you mean that secrets are encrypted in AWS Secrets Manager or when they reach Kubernetes secret?
@@DevOpsToolkit they reach Kubernetes secret encrypted. meaning, the values I see in AWS Secrets Manager are different
@@sharonlavie8210 Normally, values in k8s secrets are encoded, not encrypted. Can you confirm that's not the case? You can do it with something like `echo -n "VALUE_FROM_THE_K8S_SECRET" | base64 -d`.
@@DevOpsToolkit yes, you're right!! it is base64 encoded. just need to figure out how to decode the values i get from the secret when i use them in my gitsync sidecar.
You are #1!!
I really enjoy watching your videos and I learn a lot from you.
Will increase my support in your channel, you totally deserve it ❤
I found the solution. very simple, the username and password to github were base64 encoded, i saved them without the base64 encoding in AWS Secrets manager.
Problem solved!! yay! :)
How do you manage your secrets nowadays? Is ESO still your preference over Sealed Secrets?
That depends on whether you do use a secrets store or not. Unless you're a "small shop", using secrets store is the way to go and, in that case, ESO is, in my opinion, the best choice. Otherwise, Sealed Secrets is my choice for managing secrets without a secrets store.
@@DevOpsToolkit Thanks for your reply! I am new to all this so your content has helped a lot!! Please please continue the great work.
@@DevOpsToolkit I personally like the simplicity of sealed secrets, since there are no external dependencies and access control is only at the cluster level (or even only at the GitHub repository level if the person generating the secrets has the public key in hands, they dont even need acces to the cluster, they just encrypt the secret and push it to git). The only thing I can think of as an extra bonus for ESO is the PushSecret thing, but in my case the Platform team provided me with a cluster with Crossplane installed and all the stuff I need to deploy my application so Crossplane and my apps are the same cluster (not sure if its a good or bad idea though).
If I am in a position to choose secret manager. How can I come to trust this service?
It is not a service. It's an open source project meaning that the code is at your disposal to review it. There is nothing hidden in external secrets.
Thanks
Thanks a ton.
Where do you store your secret for ClusterSecretStore?
That's indeed a "chicken and egg problem" and I do not have a good answer for that except to say that once a secret is created in a "typical" fashion (e.g., `kubectl create secret`) it can be managed by External Secrets from there on. It's a similar problem to "how to install Argo CD without Argo CD" or "how to create a cluster where Crossplane will be running".
@@DevOpsToolkit same with ClusterAPI. WTF is going ooooooooon lol
There is always some prerequisite for any tool. We're all trying to reduce those initial dependencies but they are always there in some form or another.
@@DevOpsToolkit yes, yes, but the chicken and egg issue always makes me smile lol btw great video Victor!
Having this secret within the cluster as an opaque, is safe? I mean isn't it more safe to have these secrets only loaded in runtime by the application?
The idea, in general, is for the apps to focus on business logic and leave everything else to external processes. That applies just as much to networking, storage, and TLS as to secrets. The result is the same and the only real difference is where the logic is coded.
@@DevOpsToolkit thank you for your reply, so basically in terms of security, keep these secrets on cluster are safer than on the apps, with a plus of isolating complexity from the development. Cool
Can you please do video about Doppler I think it's much easier and more user friendly when it comes to secrets
Adding it to my to-do list...
SOPS secrets in git, only values, integrates with cloud for master key
now with PushSecret you can store secrets in the secret manager.
Yeah. That was the feature I missed the most.
Only thing I want to understand is how to combine this with Flux.
There is nothing special to do. You can push ESO manifests to git like any others and flux will sync them.
Secret Store CSI Driver does not float your boat?
I haven't used it sufficiently (yet). I've been using GoDaddy solution for a long time (External Secrets is a continuation of that project) so I haven't had the need to dive into Secret Store CSI. Is it good? Should I switch to it?
@@DevOpsToolkit Secret Store CSI Driver is a k8s standard with lots of supported integrations (e.g., ASCP is Amazon's provider). In many cases you can avoid creating k8s secrets, as per best security practice, but creating a k8s secret is also supported.
@@joebowbeer I definitely need to go back to SSC. It must have changed a lot since the last time I checked it (over a year ago).
Does it support pushing secrets to external stores? That's one of the main things missing in External Secrets (and I believe they're working on it).
Secret Store CSI Driver is great if you need to avoid having Kubernetes Native Secrets (or sensitive data in ETCD). This is specially useful with some specific compliances and Very specific sensitive workloads. Just need to remember that the service account token used by SSCSI driver is on ETCD anyways. We've seen people use both projects, even simultaneously, in the same organization. Remember: All your practices around access management, avoidance of initial attack vectors and having good identity boundaries will be way more important in avoiding people getting secrets than what tool you choose to synchronize secrets, or if they are in volumes or etcd :)
@@DevOpsToolkit They don't push secrets and I don't think it is in their roadmap, since their focus is being a Storage Interface. The focus of the SSCSID project is not to have a reconciliation loop creating/updating stuff. The provider (like ASCP mentioned here), is a privileged daemonset in the cluster, and SSCSID is just what the names tell us, a storage driver so kubernetes knows how to mount the secret as a volume in the pod. Which is why, even if you enable secret sync with SSCSID you would need to mount it first to a pod. So with image pull secrets that you usually would not use in a pod, you would have to first mount to a dummy pod so it ends up in a K8S native secret.
I think that if we believe 'each change should have a corresponding commit in Git,' then this method of secret management is not well-suited to the GitOps model. I prefer SOPS so I can handle changes to secrets like other resources. If the Git repository is the sole source of truth, then we should not allow other parties to inject changes directly into our infrastructure.
I do not think that git is the source of truth. It is the source of the desired state. The truth is the actual state which are resource at runtime. The actual state might or might not have drift when compared to the desired state.
Also, Git often contains higher level abstractions. It might have a Deployment which, at runtime, managers replica sets which manage pods which contain containers. All those are the source of truth and not only the intention to have a Deployment.
As for ESO... It is not that different from, let's say, containers. While ESO references a secret which is later pulled into a cluster, a container definition references an image which is also pulled into a cluster and converted into actual containers.
Hlo @aleskxyz...can you please share the files to implement SOPS
1password operator ..., but isn't free
We are trying to prevent people pushing their secret.yaml to git BY letting them push their secret-store.yaml to git, very wise Lol
but would we not mask the values for that secret store manifest in the pipeline library? , for example - using variable groups in Azure DevOps ¯\_(ツ)