Manage Kubernetes Secrets With External Secrets Operator (ESO)
HTML-код
- Опубликовано: 15 авг 2024
- How do we manage secrets in Kubernetes clusters? If they are stored in external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, and others, we need to pull them into the clusters. External Secrets Operator might be the solution.
#externalsecrets #kubernetes #secrets
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.co...
🔗 External Secrets Operator (ESO): external-secre...
🎬 Bitnami Sealed Secrets - How To Store Kubernetes Secrets In Git Repositories: • Bitnami Sealed Secrets...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/v... to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Courses, books, and podcasts 🚀 ▬▬▬▬▬▬
📚 Books and courses: www.devopstool...
🎤 Podcast: www.devopspara...
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Secrets In Kubernetes
04:22 Managing External Secrets
10:12 External Secrets Operator Pros And Cons 
Наука
![Chuckyy - Devil Hours [Official Music Video]](http://i.ytimg.com/vi/f3wZGj0VnZU/mqdefault.jpg)
![Finesse2Tymes & Og Boo Dirty - Real Recognize Real [Official Music Video]](http://i.ytimg.com/vi/3xb4bl5PWF8/mqdefault.jpg)
How do you manage your secrets?
Banzai vault injector
@@oftheriverinthenight I haven't tried it yet. It seems similar to Secrets Store CSI (but limited to Vault). Do you recommend it?
you also may want to use stakater/reloader to rollout restart your apps so they start using the new secret
also, keep an eye to your cloud secrets manager bill as the constant monitor will start adding costs
ArgoCD + Argo Vault Plug-in to external Vault Cluster
@@DevOpsToolkit It depends on the situation, in my curret status (migrate old apps), our first step was to put config in env vars and with Banzai Injector is pretty easy to put in vault an env var inside a gitops repo with this kind of content: ENVAR: "vault:mystorage/myteam/myitem#mykey", you can aso create secrets with another syntax ENVAR: ">>vault:mystorate/myteam/myitemykey". And can also inject iit in the middle of an envar or configmap. But External Secret Operator did not exist when I began migration so at the moment I could not evaluate what's the best solution. For sure I will try External Secret Operator when I have time.
Great Video!
PushSecrets are being developed and will soon be available! Aiming for next release, or the one after that :)
Looking forward to it!!
"If you have a problem, if no one else can help, and if you can find them, maybe you can hire the A-Team." and now you have got me the title theme of the A-Team stuck in my head! 😉🤔
Your vids are the best ... I always learn a lot from your channel.. Thanks for sharing your knowledge :D
I definitely love our external-secrets integration with Hashicorp Vault. It makes management much easier, as long as you have a single source-of-truth. It takes some getting used to, but always create/manage secrets in the external store and sync them using external-secrets. Next up: creating an operator to restart pods when a DB secret changes!
Have a look over Reloader by stakater :)
@@knelasevero Interesting, thanks!
@mtik000 Using Hashicorp Vault Injector the secrets can be updated dynamically without requiring the pod to be restarted. It monitors the Vault secret for changes and injects the new one into a specific path in the pod (templates are also supported). Works great for me.
I have also tested Secret CSI and worked in the same as Vault Injector (injecting the secret into the POD), but give also the option to create secrets resources inside K8S. But I missed the template option which is a very handy feature (maybe it does and I overlooked).
@@hutgerhauer The part I'm talking about, though, is about getting the application running inside the pod to use the new secret. Our apps generally only load the secrets on startup. I prefer not to have the secret placed anywhere on the pod itself (or in a k8s secret). The applications themselves authenticate with the Vault server using the `jwt` mounted to the pod, then read whatever secrets they need directly from Vault. I think it either scenario (yours or mine), the application needs to understand the secret has changed.
@@mtik000 got you. Well, if the application is not able to pick the newly injected secret without restarting, there's no point on using solutions such as Vault Injector or Secrets CSI. In my case, the apps do... they are able to detect a config file change and reload the settings.
I love to use Doppler to store my secrets and use Doppler CLI on pipelines for manage and use them :)
Great explanation Victor...Keep it up
At this moment I want to bring to your attention Doppler. (Not affiliated anyhow)
It is a Secret Store (SaaS) which as a service is pretty good and feature-complete. It has a great web UI, straightforward CLI, and integration with many services.
Now, Doppler has its own Kubernetes operator that does what external secrets do. You create one k8s secret with access token to Doppler and then a DopplerSecret which creates a k8s Secret. When I make a change to a secret in Doppler, it propagates immediately. On top, it has support for restarting deployments that are referencing secrets that were modified by adding one annotation. No need for Stakater/Reloader.
That being said, there’s are 3 cons I find with external secrets, compared to Sealed Secrets:
1. GitOps framework is “violated” by not being able to follow secrets stored in External Password Stores. That means there will be 2 sources of truth (git repo + Secret Store, etc) that need to be compared in conjunction to understand when some change happens to the cluster.
2. In k8s manifests files, the k8s-native secret is not created as a resource (by us). I found the lack of support of secrets in k8s tools (like the k8s plugin in Intellij) annoying, since it will not recognize the secret that the Custom Secret resource is creating. It will mark it with a squiggly line.
3. Finally, you have no way to know what kind of secrets (which keys) are stored in an external secret, without looking at it inside the cluster. Like in your video, where you have username, host, password, port. When writing your manifests, you will have to go either in your Secret Store or use kubectl to find out what exists.
Long text!! Sorry! 😅
Thanks for the recommendation. I'll check out Doppler ASAP.
1. You are right on this one, but many other resources are violating that. A reference to an image in, let's say, a Deployment. Even sealed secrets are violating that (in a way). The secret itself is not stored in Git but generated inside the cluster (by encrypting the value from the SealedSecret manifest). To be clear, I agree with you and, personally, I prefer SealedSecrets. They are more GitOps-friendly, but they also come with their own set of issues.
2. I think that's the nature of Kubernetes. We tend to create higher-level resources that a controlling the "real" ones. We create a Deployment that creates ReplicaSets, that create Pods, that manage containers. Similarly, Ingress is managing external load balancers while Crossplane or ClusterAPI are managing all sorts of external resources. We rarely manage "stuff" directly but through controllers that are in charge of managing the actual resources (often indirectly).
3. I think that SealedSecrets are not much different. Both are creating "other" resources that eventually create Kubernetes secrets. The major difference is that one (SealedSecrets) is decrypting secrets inside the cluster while the other (External Secrets) is pulling them from somewhere else (just as containers are pulling container images). In both cases, we cannot know what the secret is by inspecting the manifest. Now, if you're referring to individual fields of a secret, they can be specified with External Secrets as well. I just used a shorter version saying "give me everything because I stored everything as one big secret in GCP".
Very nice, I just deployed vault to my cluster, this came like on call :D
You save my project, thank you so much!
The 'I'm not touching my cluster' frame is meme-worthy. Just saying.. ;)
Anyhow, akuity recently did a webinar on the subject of gitops-friendly secrets management, and the external secrets operator does seem to be the "best" option.
Argocd has some plugins too.
Thanks, that was helpful!
If you have a multitenance cluster shared between different teams and you have a vault instance shared between the different teams, you would need some kind of permission management or control which team may read which endpoints from the vault. Otherwise everybody can read everything (unless that what you want?). I think that Secret store CSI driver is much more mature solution where you may define vault policy roles based on k8s namespaces and map those roles to the teams.
Hey B K! Vault Policy Roles with Secret Store CSI driver are great indeed. However, ESO supports this approach ever since version 0.1 (since the beginning ). You can create multiple SecretStores, each of them bound to a given role. I would like to invite you to please look for multitenancy setups on External Secrets Operator docs (I cant point you there since youtube will delete the comment).
I'm now using external Secrets Operator because sealed secrets failed me big time :-( I upgraded k8s version in a cluster using sealed secrets, and after the upgrade I could not unseal the secrets anymore... I know, it should not happen but it did :-( I had to recreate all sealed secrets again (no i did not had a backup... but since the issue it seems was the k8s cluster certificate change not sure if it would have helped ). Has this happened to anyone ? (Hint: only seems an issue when using strict... at least with cluster-wide it never happened.)
thanks, surely useful for companies with centralized credentials.
I still mess up things with Vault :)), I hope I will master it in one day
Thank you very much for the explanation. It seems to be a reverse crossplane for secrets 😅..
Actually, crossplane is one of the reasons I explored external secrets. I need something to pull secrets into my clusters so that crossplane can use them but also something to push secrets generated by crossplane into my secret store. That's why I mentioned "secrets push" as the only missing feature. Once that is available, the secrets generated with crossplane will be available wherever I need them.
@@DevOpsToolkit I'm very curious about how you combined Crossplane with External-secrets.
I generate kubernetes clusters with Crossplane and get the kubeconfig for them in crossplane-system namespace.
I tried with Sealed Secrets but passing the private certificate to decrypt secrets around different clusters is a pain.
The feature to push secrets is now available with External Secrets, so what would the approach be?
Just using a kubernetes provider and push the secret from the management cluster into the created clusters?
@@IvanRizzante Getting secrets into k8s is easy. Pushing them from k8s to a secret store is a challenge. A while ago, I was urging ESO community to add "push" feature to the project and they did that. However, I did not yet have enough time to dive into it.
Hi Victor! Amazing video. Thanks! Just one question: what do you think about argo vault plugin? Is good? Or is it better to use external secrets?
I haven't tried it yet. In general, I do not think that projects like Argo should be concerned with secrets. We already have solutions that manage them well and I'm sceptical whether side projects like that can out as much attention to it as those fully focused on secrets like external secrets, sealed secrets, and others. Nevertheless, let me check it out before saying anything about it.
Victor, did you already consider a video on (cloud provider agnostic) secret stores? I would really love to see one from you since you are focusing on K8s and just do awesome content
I do not think there is anything better than ESO right now. The only alternative that i am aware of is SealedSecrets and SSCSID. SealedSecrets does not work with secret stores like vault and i am not sure that SSCSID direction is a good one for all use cases. If you know of any other solution, i would be very happy to give it a spin.
@@DevOpsToolkit thanks for the input! I really like ESO, it rather is Vault that gets on my nerves.
If I find out about something else, I will let you know but I guess you will be the first one to know, anyway 😄
I just realized that you asked about secret stores and not secret managers. So, my previous answer must have been confusing.
For secret stored, Vault and those provided by hyperscalers are the only ones i see being widely used and I haven't seen any challenger on the horizon 😔
Hi Victor, would you recommend some tool to keep a track of running services / ports / IPs etc... on Kubernetes cluster... something like "live" documentation, if anything like that even exists... I know of Backstage from Spotify but the deployment of that is kind of strange. I'm looking for something I can deploy to Kubernetes, and people can go there, look up app running on the cluster, find the owner, some documentation to it and so on...
I like using Rancher to get an overview and use labels as well as annotations (plus Rancher projects that group namespaces) for organization.
My answer will be biased... I tend to use crossplane which, through compositions, gives me different ways to groups resources and expose them through CLI/UI.
now with PushSecret you can store secrets in the secret manager.
Yeah. That was the feature I missed the most.
I was expecting something about Secret Store CSI Driver, but External Secrets Operator is also interesting. But what happens on the application side when the secret is updated?
I already have as csi on my TODO list... :)
Can you do a review on doppler? Both with the k8s operator and the cli approach (see their docs)
Adding it to my TODO list... :)
As a side note, I think we had someone from Doppler on www.devopsparadox.com
Any idea how this compares to the CSI Secrets Driver? It looks almost the same.
The major difference is that it does not create kubernetes secrets but mounts volumes which should be more secure (kubernetes secrets are very insecure). On the other hand, that is not a common approach in kubernetes and might cause issues with third party tools.
Thanks
Thanks a ton.
The ESO docs for Hashicorp Vault mention that "Vault itself implements lots of different secret engines, as of now we only support the KV Secrets Engine." Did you manage to make it work with any other Secret Engine except KV? I find this pretty limiting as I have to implement dynamic secrets and secret rotation myself although Vault offers it for most if its engines.
I used it mostly with Google servers and a bit with aws secrets manager so i don't hwveuch experience with it combined with vault 😔
Hey man, great video! Helped me understand ESO a lot better :) There is one thing I'm struggling to find out though, and I'm hoping you might have an answer for me:
If I want to have keys managed by Azure Key Vault to auto-rotate every X days, and I use ESO to fetch these auto-rotating keys (with a refresh interval of 1h), then what would happen if
- ESO refreshes the secrets from Azure Key Vault
- Azure Key Vault rotates my key at scheduled time, which happens to be 1 minute after the refresh from ESO
Will my applications that use this key now not work for 59 minutes?
I've really been trying to figure this one out, but there seems to be absolutely no answer to this anywhere online. :(
When you rotate keys, you still need to keep the old key operational for a while. Otherwise, no solution would work. For example, let's say that you have authentication to a db and you rotate it. If that would mean that from there on only the new auth works, all currently established connections would fail.
I've been overlaying resources rather than values. It might be possible to overlay values as well, but I haven't tried that option, mostly because I believe that overlaying resources is closer to the "Kustomize spirit".
@@DevOpsToolkit thank you for the content and explanation Victor. Could you make a video for this specific scenario of "secrets rotation + keeping the old value" to avoid donwtime until the app gets the new value? And also when we have N replicas of the pod using that secret.
@thiagoscodeler5152 adding it to my to-do list...
Can it wildcard the secret names and bring in all the secrets from the source. I don't want to specify them by name. I want to selectively override certain default values from the app config, but I want to leave out the secrets from my vault that I don't care about.
As far as I know, it cannot use wildcards. You need to be specific which secrets you want.
@@DevOpsToolkit Right, that's what I thought. What I'm focusing on right now is app configuration, which needs about 10 or so values, some of which are sensitive, some not. They all have good defaults, so if you don't give a value, the app uses the default. So what I'm thinking is I only need one secret in Kubernetes ... the connection string that gets them to Hashicorp Vault, Azure Key Vault, AWS Secrets Manager, etc. All other config settings go in the external vault. That way they can change config values after-the-fact and they don't need to keep going back to their Helm charts or K8s manifests. It would be a real hassle for the user to need to keep going back to k8s just to change one simple config value.
So if I only need to store one secret, then external-secrets isn't that much value. Sounds like more trouble than it is worth.
Hi Victor, thanks for another interesting video! How does the complexity of configuring external secrets fair vs bitnami sealed secrets? My team is transitioning to k8s, so we're looking for solutions that are easy to install / manage and would give us immediate benefit
In case of External Secrets, the complexity is in managing secrets in an external store (e.g., HashiCorp vault) as well as maintenance of the store itself (if it's not SaaS). If we ignore that, External Secrets are pretty easy and straightforward. SealedSecrets is easy as well and there's nothing to manage but secrets themselves stored in Git.
Both are great and neither is overly complex. The main question is whether you prefer to keep secrets in an external store or in Git (an encrypted version of them).
@@DevOpsToolkit awesome, thank you for explaining
Thank you for all your video content. This is something I'm looking to implement on our cluster soon.
I have one question though. When it comes to storing trustore.ts files as secrets, how would we do this with AWS Secrets Manager. Should we base64 encode it? Any help would be greatly appreciated.
All secrets in Kubernetes are base64 encoded, so yes. You would need to create that secret first either with kubectl or anything else. However, once the secret is there, you can manage it with external secrets.
Ok. thank you for your help.
Can you please do video about Doppler I think it's much easier and more user friendly when it comes to secrets
Adding it to my to-do list...
Where do you store your secret for ClusterSecretStore?
That's indeed a "chicken and egg problem" and I do not have a good answer for that except to say that once a secret is created in a "typical" fashion (e.g., `kubectl create secret`) it can be managed by External Secrets from there on. It's a similar problem to "how to install Argo CD without Argo CD" or "how to create a cluster where Crossplane will be running".
@@DevOpsToolkit same with ClusterAPI. WTF is going ooooooooon lol
There is always some prerequisite for any tool. We're all trying to reduce those initial dependencies but they are always there in some form or another.
@@DevOpsToolkit yes, yes, but the chicken and egg issue always makes me smile lol btw great video Victor!
SOPS secrets in git, only values, integrates with cloud for master key
This is great to showcase operator outreach however I am not still convinced what happens if the cluster is exposed by some chance as we are syncing secrets from external secret management services which essentially syncing the external secret object(copy) into Kuebernetes cluster and making it available as another dumb Kuebernetes secret anyone can decode with base64 -d. Sorry, in case I have misconstrued the video and topic. I am more interested how ESO handles encryption at rest not just the plain naked secrets stored in SM.
ESOs responsibility stops when secrets in kubernetes are created. From there on, it all depends on permissions you defined, whether etcd is encryped, etc. The alternative is bot to use kubernetes secrets at all. I will publish a video on that subject in 3-4 weeks. TL;DR; it is close to impossible not to use kubernetes secrets.
I was watching another intro video about External Secrets and the presenter said that Sealed Secrets doesn't scale well, but he didn't point out why. I'm wondering why he said that (and rereading this comment, I might have answered my own question).
The missing secrets generation and store to secret stores point you make is also why I feel Sealed Secrets are a "better fit" for what we want to accomplish. However, theoretically, we might have a large number of clusters, so that scaling comment does raise concerns.
Correct me if I am wrong, but if the generation of the sealed secret and (controlled) storage of it in a git repo along with the generation of the values to be encrypted themselves are all automated (i.e. no manual sealed secret generation), then what is to stop the process from scaling out to 1000's of clusters, when each cluster has its own sealed secret controller? This is also assuming the apps using the sealed secrets are cluster specific too. I realize, for apps that might be used across large amounts of clusters, sealed secrets wouldn't scale well. Is that what he meant? For that though, I'd use a different pattern/ tool, maybe even ESO?
Btw, I also looked at the ESO docs and it doesn't seem to have a method to store back to the external secret stores yet. How can the generation of secrets even be automated then? To me, that is a real scalability issue.
I'm not sure what he meant when saying that sealed secrets do not scale. The job is relatively easy and is performed only when manifests change to there is no constant demand for resources. Also, i haven't experienced any performance/scaling issues myself.
@@DevOpsToolkit Awesome. Appreciate the quick reply. From all the solutions out there, sealed secrets fits best for what we require. I am also now liking the idea more and more to store the app "target" state in the clients' own repos.
thanks for the video.
I'm using AWS and by default the secrets are encrypted and then the secret is not usable (trying to use it with Airflow GitSync side cart)
Could you suggest a solution for my problem?
Do you mean that secrets are encrypted in AWS Secrets Manager or when they reach Kubernetes secret?
@@DevOpsToolkit they reach Kubernetes secret encrypted. meaning, the values I see in AWS Secrets Manager are different
@@sharonlavie8210 Normally, values in k8s secrets are encoded, not encrypted. Can you confirm that's not the case? You can do it with something like `echo -n "VALUE_FROM_THE_K8S_SECRET" | base64 -d`.
@@DevOpsToolkit yes, you're right!! it is base64 encoded. just need to figure out how to decode the values i get from the secret when i use them in my gitsync sidecar.
You are #1!!
I really enjoy watching your videos and I learn a lot from you.
Will increase my support in your channel, you totally deserve it ❤
I found the solution. very simple, the username and password to github were base64 encoded, i saved them without the base64 encoding in AWS Secrets manager.
Problem solved!! yay! :)
If I am in a position to choose secret manager. How can I come to trust this service?
It is not a service. It's an open source project meaning that the code is at your disposal to review it. There is nothing hidden in external secrets.
Having this secret within the cluster as an opaque, is safe? I mean isn't it more safe to have these secrets only loaded in runtime by the application?
The idea, in general, is for the apps to focus on business logic and leave everything else to external processes. That applies just as much to networking, storage, and TLS as to secrets. The result is the same and the only real difference is where the logic is coded.
@@DevOpsToolkit thank you for your reply, so basically in terms of security, keep these secrets on cluster are safer than on the apps, with a plus of isolating complexity from the development. Cool
Only thing I want to understand is how to combine this with Flux.
There is nothing special to do. You can push ESO manifests to git like any others and flux will sync them.
I think that if we believe 'each change should have a corresponding commit in Git,' then this method of secret management is not well-suited to the GitOps model. I prefer SOPS so I can handle changes to secrets like other resources. If the Git repository is the sole source of truth, then we should not allow other parties to inject changes directly into our infrastructure.
I do not think that git is the source of truth. It is the source of the desired state. The truth is the actual state which are resource at runtime. The actual state might or might not have drift when compared to the desired state.
Also, Git often contains higher level abstractions. It might have a Deployment which, at runtime, managers replica sets which manage pods which contain containers. All those are the source of truth and not only the intention to have a Deployment.
As for ESO... It is not that different from, let's say, containers. While ESO references a secret which is later pulled into a cluster, a container definition references an image which is also pulled into a cluster and converted into actual containers.
Secret Store CSI Driver does not float your boat?
I haven't used it sufficiently (yet). I've been using GoDaddy solution for a long time (External Secrets is a continuation of that project) so I haven't had the need to dive into Secret Store CSI. Is it good? Should I switch to it?
@@DevOpsToolkit Secret Store CSI Driver is a k8s standard with lots of supported integrations (e.g., ASCP is Amazon's provider). In many cases you can avoid creating k8s secrets, as per best security practice, but creating a k8s secret is also supported.
@@joebowbeer I definitely need to go back to SSC. It must have changed a lot since the last time I checked it (over a year ago).
Does it support pushing secrets to external stores? That's one of the main things missing in External Secrets (and I believe they're working on it).
Secret Store CSI Driver is great if you need to avoid having Kubernetes Native Secrets (or sensitive data in ETCD). This is specially useful with some specific compliances and Very specific sensitive workloads. Just need to remember that the service account token used by SSCSI driver is on ETCD anyways. We've seen people use both projects, even simultaneously, in the same organization. Remember: All your practices around access management, avoidance of initial attack vectors and having good identity boundaries will be way more important in avoiding people getting secrets than what tool you choose to synchronize secrets, or if they are in volumes or etcd :)
@@DevOpsToolkit They don't push secrets and I don't think it is in their roadmap, since their focus is being a Storage Interface. The focus of the SSCSID project is not to have a reconciliation loop creating/updating stuff. The provider (like ASCP mentioned here), is a privileged daemonset in the cluster, and SSCSID is just what the names tell us, a storage driver so kubernetes knows how to mount the secret as a volume in the pod. Which is why, even if you enable secret sync with SSCSID you would need to mount it first to a pod. So with image pull secrets that you usually would not use in a pod, you would have to first mount to a dummy pod so it ends up in a K8S native secret.
1password operator ..., but isn't free
We are trying to prevent people pushing their secret.yaml to git BY letting them push their secret-store.yaml to git, very wise Lol
but would we not mask the values for that secret store manifest in the pipeline library? , for example - using variable groups in Azure DevOps ¯\_(ツ)