We moved away from sealed secrets to ESO recently, it seems easier to use with all our helm charts as compared to CSI driver. I have a dream one day ESO (or any secret solution actually) will also restart pods when a secret is rotated 🤞
@@pietersmit621 True. SealedSecrets and SOPS are my personal favorites. Nevertheless, what makes SSCSID interesting is that there are no Kubernetes secrets in the cluster. Now, whether that is a good idea (no matter the implementation) is a different question.
We use: sealed secrets for things that pre-date our Vault cluster and bootstrap things (i.e. the Vault cluster is down), and ESO for cluster-things (e.g. AltertManager Slack webhooks). All of our apps read directly from Vault. I would imagine that we might move to AWS Secrets Manager in the future, and maybe SSCSID is a more agnostic way to go?
I use this at work a lot. Biggest con I've found is that you HAVE to mount your SecretProviderClass to a pod for secrets to created in the cluster. This means if you have a CRD that needs a secret but no real pod that goes along with it then you need to run a small busybox pod to inject the secret.
I agree, but I would generalize it a bit and say that it can cause problems or extra work when using it with third-party resources, no matter whether those are CRDs, Helm charts designed to use only secrets, etc.
Thank Victor for sharing this! Always good to see alternatives reagarding secret managment in Kubernetes! k8s secrets should have been like ingress, or gateway API: An Interface to whatever implementation I want to use.
I think that's where Kubernetes is going. It started with "basic" implementations of most of the things we need (Deployment, Ingress, Storage, Service, etc.). Those are only simple implementations that should be extended or reimplemented. That's how we got service meshes and are not getting a standard interface for them (Gateway API). Something similar will happen to secrets. But, until it does, we need to work with what we have.
JFYI the Azure Key Vault Provider for Secrets Store CSI Driver also supports setting environment variables by syncing a kubernetes secret, not only volume mounts. Despite this, a drawback is that the k8s secret needs to be attached to a pod, otherwise the secret will be deleted. So you might need to artifiacilly attach the secret to a pod (i.e: you might need to mount the secret in the ingress controller pod to be able to use it in an ingress resource).
I think that's not specific to Azure but all providers. The problem is that once you start creating secrets as well, SSCSID loses its appeal and the reason for existence. In those cases, I would go with External Secrets Operator (ruclips.net/video/SyRZe5YVCVk/видео.html).
...and here I am and I can't decide between secrets in ENV vars, ESO, the apps reading directly from the store, and now I have to think about SSCSID! so many choices.
Oh wow - Mr. rpicluster himself. Tells me watching this channel is not a waste of time (ok, knew that before) and gives me the chance to say thanks for the super tutorial! Coffee due once through.
I'm not sure I understood the question. Are you trying to use secrets as Helm values? If that's the case, don't. If an app needs info available as a secret, that app should mount that secret.
Great video, thanks! I must be missing something about how the secrets are mounted for third party apps. You say that this may not work for third party apps that use Kubernetes Secrets, but it's my understanding that inside th ePod, the K8s secret (when mounted as a volume) would look the same as a SSCSID secret as they both appear as a file with the secret as the contents. Obviously the Helm chart/manifest for that app would need modifications due to the existing use of K8s secrets, so maybe that is all you meant? And of course, if the existing app is using a K8s secret mounted as an env var then I completely get the difference. I'm just talking the scenario where K8s secrets are mounted as volumes.
As you said, helm charts would need to modified and that would result in maintenance issues. The modification would effectively be a fork and you'd need to watch for future releases of the original chart and apply the changes to the fork. Alternative would be to keep using the original chart by combining it with some sort of overlaying with, let's say, ytt, but that would also increase the complexity. In other words, it's doable, but at a price.
@@DevOpsToolkit oh I forgot to ask, was I correct in my understanding that the volume mounting mechanism looks the same inside the pod? (When not using env vars for K8s secrets)
I'm a bit... confused, in case of it's integration with Hashicorp Vault. I understand it's use case with GCP/AWS/Azure however Hashicorp Vault already mounts secrets as volume (as init or sidecar container) and supports live reloading of data. So, do you see any advantage of using SSCSID with Vault instead of using plain Vault?
Vault has quite a few ways to deal with secrets, some done by HashiCorp itself, while others by third-party solutions. Are you referring to www.vaultproject.io/docs/platform/k8s/injector?
@@DevOpsToolkit yes, I'm using it at work, except some issues with JWT validation change in Kubernetes 1.22 (or 1.21?) it works like a charm. The biggest advantage I see is that you just need to use proper annotations to injects secrets (so Helm charts that allow setting annotations from values would work). It is even possible to load secrets as environment variables with some hack but that hack should also work for this solution.
@@krzysztofwiatrzyk4260 Vault is the most advanced secrets store with decently big ecosystem so there is no big reason to use ESO or SSCSID. You can accomplish similar results with Vault Injector (similar to ESO) or Vault CSI (similar to SSCSID). The major difference is that ESO and SSCSI are more in line with the vault Kubernetes expects us to define things (through CR instead of annotations) and with more decoupling between the app (or app manifest) and secrets. There are other differences. Nevertheless, those might be significant enough to warrant a switch from Injector to ESO or Vault CSI to SSCSID. That being said, there will be more differences in the future. HashiCorp tends to ignore or go against Kubernetes so its slowly falling behind the "Kube Ecosystem". That can probably be attributed to HashiCorp not treating open source as it should, other software vendors looking to reduce the dependency on HashiCorp, their insistance on keeping Nomad alive, etc.
Hello Sir...Can you please guide me to retrieve the AWS secrets and store it to kubernetes pod/container.using helm charts? your quick reply help me a lot ..!! Thank you so much in advance
If the secret already exists, all you have to do is mount it to the pod. Once you do that, all entries of the secret will be available as environment variables.
How sir, for example if Mount means as file it will come and sit in the location where I've chosen. But I didn't say anything in pod spec about env...?
@@ramallways6321 If you're talking about Kuberentes secrets, than you can specify which secret entry becomes an env. variable through `secretKeyRef` (kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-a-container-environment-variable-with-data-from-a-single-secret).
I'm not talking about k8s secret. For example if Mount my Azure secret into my pod via secret provider class. Then all secrets will come and sit like file in my pod. My question here is how this secret will be available in pod's env?
My bad. I misunderstood... With ESO you get only a file and the container that mounts it would need to convert it into environment variables. I think that there is also an option to create kubernetes secrets as well but I haven't used it much since that would defy the purpose of using ESO.
That would result in the destruction of the pod where the volume is mounted and the creation of a new one without the volume. Out of curiosity... Why would you remove the volume?
You can, you need to set it to create a k8s secret in your SecretProviderClass and then mount it as a volume in your pod. It will create the k8s secret and you pull from there with envFrom. Not ideal but it does work
Thanks for your video, when working with secrets one of the main use cases is, that secrets can change during lifetime (f.e. secret rotation) most applications read the secrets/config in during startup, but don't update it anymore. The way I work around this problem is stakater Reloader to auto restart on config/secret change. How can Secrets Store CSI Driver achieve this use case?
Enable Auto Rotation of Secrets to refresh, but if you need to be notified then you'll need to implement a watcher, e.g., of the file contents of the volume mount of the CSI Driver.
From the app perspective, it's the same. Secrets are available as files and your app needs to reload itself when they change. The major difference is that SSCSID loads those files by mounting volumes instead of mounting secrets (which are files as well, but are also stored in etcd).
Not the same use case. External secret pulls from third party and creates a secret inside the cluster. Here you never create any sensitive data inside the cluster
@@shulyakav if security is done right you can minimize the attack surface by creating multiple service accounts for multiple purposes. If someone gets to your cluster , it’s going to be with some kind of cluster role or role attached to it. If you have secrets in a namespaces they are easier to grab than connecting to a pod since the pod/exec permission is not very common except cluster admins such as devops and product security. I am not saying it’s a 100% covering all soliton but it does minimize the attack surface
I think you'd need to read that file and convert into env. vars. As far as I know, SSCSID does not do that and you'd either need to create your own process (in a sidecar) or look for some tool that does that. P.S. I'm ignoring the fact that SSCSID can create secrets as well (and thus provider envs. vars.) since that defies its purpose (to not have secrets).
@@DevOpsToolkit as common as a need as this is I'm surprised I've not found an established and popular method that works with any of pod setup. All I ever see is people just say well you can do it and no actual useful information about how
@@travissobeck4939 Assuming that the file is in key/value format, a simple `source my-file` should do. You can put it into the command of the container (e.g., `source my-file && my-app`).
There is a caveat i want to add with csi. If ur syncing certs as k8s tls secrets. It can only create tls.crt and tls.key but it cant create ca.crt as secret data.
How do you edit your videos, its awsm and what tool do you use to record your session, please let me know, also if possible please create a video on the same as well. it will be a good learning experience for me and for others.
@@devopsbucket2549 Here it goes: * Screen recording: OBS * Video recording: Sony ZV-E10 with Sigma 16 mm Lens * Audio recording: Rode NTG4 (for videos) and ElGato Wave (for streams) * Lightning: 2 ElGato Key Lights in front and LifX lightbulbs in the back (RGB) * Teleprompter: Desview T3 * Computer: iMac (will be moving to Mac Studio in the future)
That's true. I recoded it in a different location that has terrible echo. It's a temporary place where I go during summer and that I did not yet sound proof.
That is indeed true. `hostPath` is required. Now, I would not go as far as deny anything to use `hostPath`. Instead, like with almost anything else, I'd make sure that only certain processes can use it in a controlled manner. Otherwise, you'd have trouble running quite a few other tools effectively (e.g., Tekton, Argo Workflows, etc.).
@@DevOpsToolkit Security guys are adopting (learning) Kubernetes and the security tools around Kubernetes. They want to tick as many boxes as they can :D
@@JaydeepDave12 That's true and, given everything else being equal, I'd pick a tool that does not mount `hostPath`. I still recommend External Secrets Operator over SSCSID for the cases where secrets are in external secrets manager.
What do you use to manage Kubernetes secrets? External Secrets Operator, Sealed Secrets, Secrets Store CSI Driver, or something else?
We moved away from sealed secrets to ESO recently, it seems easier to use with all our helm charts as compared to CSI driver. I have a dream one day ESO (or any secret solution actually) will also restart pods when a secret is rotated 🤞
@@Luther_Luffeigh A feature to reload (not restart) Pods is coming to Kubernetes :)
Mozilla sops encrypted values in git
@@pietersmit621 True. SealedSecrets and SOPS are my personal favorites. Nevertheless, what makes SSCSID interesting is that there are no Kubernetes secrets in the cluster. Now, whether that is a good idea (no matter the implementation) is a different question.
We use: sealed secrets for things that pre-date our Vault cluster and bootstrap things (i.e. the Vault cluster is down), and ESO for cluster-things (e.g. AltertManager Slack webhooks). All of our apps read directly from Vault. I would imagine that we might move to AWS Secrets Manager in the future, and maybe SSCSID is a more agnostic way to go?
I use this at work a lot. Biggest con I've found is that you HAVE to mount your SecretProviderClass to a pod for secrets to created in the cluster. This means if you have a CRD that needs a secret but no real pod that goes along with it then you need to run a small busybox pod to inject the secret.
I agree, but I would generalize it a bit and say that it can cause problems or extra work when using it with third-party resources, no matter whether those are CRDs, Helm charts designed to use only secrets, etc.
nails match the background. 😄
Thank Victor for sharing this! Always good to see alternatives reagarding secret managment in Kubernetes!
k8s secrets should have been like ingress, or gateway API: An Interface to whatever implementation I want to use.
I think that's where Kubernetes is going. It started with "basic" implementations of most of the things we need (Deployment, Ingress, Storage, Service, etc.). Those are only simple implementations that should be extended or reimplemented. That's how we got service meshes and are not getting a standard interface for them (Gateway API). Something similar will happen to secrets. But, until it does, we need to work with what we have.
Probably another negative thing is the fargate profile + CSI storage driver , it does not work all together, due to daemon set
JFYI the Azure Key Vault Provider for Secrets Store CSI Driver also supports setting environment variables by syncing a kubernetes secret, not only volume mounts.
Despite this, a drawback is that the k8s secret needs to be attached to a pod, otherwise the secret will be deleted. So you might need to artifiacilly attach the secret to a pod (i.e: you might need to mount the secret in the ingress controller pod to be able to use it in an ingress resource).
I think that's not specific to Azure but all providers.
The problem is that once you start creating secrets as well, SSCSID loses its appeal and the reason for existence. In those cases, I would go with External Secrets Operator (ruclips.net/video/SyRZe5YVCVk/видео.html).
...and here I am and I can't decide between secrets in ENV vars, ESO, the apps reading directly from the store, and now I have to think about SSCSID! so many choices.
That's the blessing and the curse of the ecosystem's size and speed of innovation. There are too many choices.
Using both External Secrets Operator and Flux's built-in SOPS decryption (which is similar to Sealed Secrets, but uses SOPS, and is native to Flux).
I think this is how OpenFaaS is doing it, it also provides secrets inside the pod where the function run as files.
Oh wow - Mr. rpicluster himself. Tells me watching this channel is not a waste of time (ok, knew that before) and gives me the chance to say thanks for the super tutorial!
Coffee due once through.
@@armynyus9123 this channel is awesome ! I would recommend it in heart beat.
Thanks! Looking to know more about this use case.
Thanks, Victor!
Exciting topic as always.
Cute nails by the way 😀
Iam using helm values to deploy all manifest, and all the secrets are stated on that (plain text) How can implement this to my values?
I'm not sure I understood the question. Are you trying to use secrets as Helm values? If that's the case, don't. If an app needs info available as a secret, that app should mount that secret.
Great video, thanks!
I must be missing something about how the secrets are mounted for third party apps. You say that this may not work for third party apps that use Kubernetes Secrets, but it's my understanding that inside th ePod, the K8s secret (when mounted as a volume) would look the same as a SSCSID secret as they both appear as a file with the secret as the contents.
Obviously the Helm chart/manifest for that app would need modifications due to the existing use of K8s secrets, so maybe that is all you meant?
And of course, if the existing app is using a K8s secret mounted as an env var then I completely get the difference. I'm just talking the scenario where K8s secrets are mounted as volumes.
As you said, helm charts would need to modified and that would result in maintenance issues. The modification would effectively be a fork and you'd need to watch for future releases of the original chart and apply the changes to the fork. Alternative would be to keep using the original chart by combining it with some sort of overlaying with, let's say, ytt, but that would also increase the complexity. In other words, it's doable, but at a price.
@@DevOpsToolkit Got it. Thanks for clarifying!
@@DevOpsToolkit oh I forgot to ask, was I correct in my understanding that the volume mounting mechanism looks the same inside the pod? (When not using env vars for K8s secrets)
@@Tomsta17 Yes. In both cases, filesystem is mounted into containers and secrets are available as files (ignoring env. vars.).
Can I create Env variables without creating secrets? Almost all apps I use rely on env variables.
You can add env variables to containers through secrets, config maps, or directly in pod manifests.
I'm a bit... confused, in case of it's integration with Hashicorp Vault. I understand it's use case with GCP/AWS/Azure however Hashicorp Vault already mounts secrets as volume (as init or sidecar container) and supports live reloading of data. So, do you see any advantage of using SSCSID with Vault instead of using plain Vault?
Vault has quite a few ways to deal with secrets, some done by HashiCorp itself, while others by third-party solutions. Are you referring to www.vaultproject.io/docs/platform/k8s/injector?
@@DevOpsToolkit yes, I'm using it at work, except some issues with JWT validation change in Kubernetes 1.22 (or 1.21?) it works like a charm. The biggest advantage I see is that you just need to use proper annotations to injects secrets (so Helm charts that allow setting annotations from values would work). It is even possible to load secrets as environment variables with some hack but that hack should also work for this solution.
@@krzysztofwiatrzyk4260 Vault is the most advanced secrets store with decently big ecosystem so there is no big reason to use ESO or SSCSID. You can accomplish similar results with Vault Injector (similar to ESO) or Vault CSI (similar to SSCSID). The major difference is that ESO and SSCSI are more in line with the vault Kubernetes expects us to define things (through CR instead of annotations) and with more decoupling between the app (or app manifest) and secrets. There are other differences. Nevertheless, those might be significant enough to warrant a switch from Injector to ESO or Vault CSI to SSCSID.
That being said, there will be more differences in the future. HashiCorp tends to ignore or go against Kubernetes so its slowly falling behind the "Kube Ecosystem". That can probably be attributed to HashiCorp not treating open source as it should, other software vendors looking to reduce the dependency on HashiCorp, their insistance on keeping Nomad alive, etc.
Hello Sir...Can you please guide me to retrieve the AWS secrets and store it to kubernetes pod/container.using helm charts? your quick reply help me a lot ..!! Thank you so much in advance
The gist with the commands i used in that video is in the description. You should be able to replicate what i did in aws with those.
How to set an environmental variable from third party secrets to the container instead mount as volume?
If the secret already exists, all you have to do is mount it to the pod. Once you do that, all entries of the secret will be available as environment variables.
How sir, for example if Mount means as file it will come and sit in the location where I've chosen. But I didn't say anything in pod spec about env...?
@@ramallways6321 If you're talking about Kuberentes secrets, than you can specify which secret entry becomes an env. variable through `secretKeyRef` (kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-a-container-environment-variable-with-data-from-a-single-secret).
I'm not talking about k8s secret. For example if Mount my Azure secret into my pod via secret provider class. Then all secrets will come and sit like file in my pod. My question here is how this secret will be available in pod's env?
My bad. I misunderstood... With ESO you get only a file and the container that mounts it would need to convert it into environment variables. I think that there is also an option to create kubernetes secrets as well but I haven't used it much since that would defy the purpose of using ESO.
can we unmount CSI volume as once secrets are pulled in Application ?
That would result in the destruction of the pod where the volume is mounted and the creation of a new one without the volume.
Out of curiosity... Why would you remove the volume?
The main problem I have with this is that I can’t inject the secret as an environment variable…
You can, you need to set it to create a k8s secret in your SecretProviderClass and then mount it as a volume in your pod. It will create the k8s secret and you pull from there with envFrom. Not ideal but it does work
Fix: SSCSID volume mounts can be projected to k8s secrets and any k8s secret can be injected as an environment variable
That's true. However, that goes against the main reason for existence of SSCSID (no secrets).
Should it possible to inject secret as env varible without creating k8s secrets??
🤠 Great video!!
Thanks for your video, when working with secrets one of the main use cases is, that secrets can change during lifetime (f.e. secret rotation) most applications read the secrets/config in during startup, but don't update it anymore. The way I work around this problem is stakater Reloader to auto restart on config/secret change. How can Secrets Store CSI Driver achieve this use case?
Enable Auto Rotation of Secrets to refresh, but if you need to be notified then you'll need to implement a watcher, e.g., of the file contents of the volume mount of the CSI Driver.
From the app perspective, it's the same. Secrets are available as files and your app needs to reload itself when they change. The major difference is that SSCSID loads those files by mounting volumes instead of mounting secrets (which are files as well, but are also stored in etcd).
@@DevOpsToolkit Yes exactly and so changes inside are not detected and need to be implemented as sidecar or inside the application :/
External Secrets Operator is definitely better than Secrets Store CSI Driver.
Not the same use case. External secret pulls from third party and creates a secret inside the cluster. Here you never create any sensitive data inside the cluster
@@mopsik4ever true. But anyway you will create volume with secrets inside a cluster (pod).
ESO can also used to create a k8s secret store, so you can have it fetch from either source
@@shulyakav if security is done right you can minimize the attack surface by creating multiple service accounts for multiple purposes. If someone gets to your cluster , it’s going to be with some kind of cluster role or role attached to it. If you have secrets in a namespaces they are easier to grab than connecting to a pod since the pod/exec permission is not very common except cluster admins such as devops and product security. I am not saying it’s a 100% covering all soliton but it does minimize the attack surface
million dollar question, how do I get the secrets from the file to be set as environmental variables in cases where the application just requires it
I think you'd need to read that file and convert into env. vars. As far as I know, SSCSID does not do that and you'd either need to create your own process (in a sidecar) or look for some tool that does that.
P.S. I'm ignoring the fact that SSCSID can create secrets as well (and thus provider envs. vars.) since that defies its purpose (to not have secrets).
@@DevOpsToolkit as common as a need as this is I'm surprised I've not found an established and popular method that works with any of pod setup. All I ever see is people just say well you can do it and no actual useful information about how
@@travissobeck4939 I guess that the need is not that big. Most people use Kubernetes secrets or configmaps which do provide env. vars.
@@travissobeck4939 Assuming that the file is in key/value format, a simple `source my-file` should do. You can put it into the command of the container (e.g., `source my-file && my-app`).
There is a caveat i want to add with csi. If ur syncing certs as k8s tls secrets. It can only create tls.crt and tls.key but it cant create ca.crt as secret data.
Use service mesh for internal certs and cert manager for external facing ones. Do not mount certs yourself into apps.
How do you edit your videos, its awsm and what tool do you use to record your session, please let me know, also if possible please create a video on the same as well. it will be a good learning experience for me and for others.
I record screen, video, and sound separately and send it to an agency that does the editing, animations, banners, etc.
@@DevOpsToolkit mac book screen recorder? what about lighting and all, thanks for answering
@@devopsbucket2549 Here it goes:
* Screen recording: OBS
* Video recording: Sony ZV-E10 with Sigma 16 mm Lens
* Audio recording: Rode NTG4 (for videos) and ElGato Wave (for streams)
* Lightning: 2 ElGato Key Lights in front and LifX lightbulbs in the back (RGB)
* Teleprompter: Desview T3
* Computer: iMac (will be moving to Mac Studio in the future)
@@DevOpsToolkit Thanks for answering it help me a lot.
It seems the video has a huge echo.
That's true. I recoded it in a different location that has terrible echo. It's a temporary place where I go during summer and that I did not yet sound proof.
I was told that, the secret store needs a `hostPath` mount to work!! Is that true? Production clusters don't allow to use hostPath
That is indeed true. `hostPath` is required. Now, I would not go as far as deny anything to use `hostPath`. Instead, like with almost anything else, I'd make sure that only certain processes can use it in a controlled manner. Otherwise, you'd have trouble running quite a few other tools effectively (e.g., Tekton, Argo Workflows, etc.).
@@DevOpsToolkit Security guys are adopting (learning) Kubernetes and the security tools around Kubernetes. They want to tick as many boxes as they can :D
@@JaydeepDave12 That's true and, given everything else being equal, I'd pick a tool that does not mount `hostPath`.
I still recommend External Secrets Operator over SSCSID for the cases where secrets are in external secrets manager.
Covered in the SSCSID best practices guide.
You should have asked your daughter to paint them grey!
I'm afraid that it'll be pink or green the next time.
@@DevOpsToolkit I just thought you went emo.