not everyone here is a software engineer some folks wants to get into devops r cka soo next time when u choose a topiic like this plz make it as like for beginners to advanced level
That's a good suggestion. I'll do my best to either start earlier in a story or link to other videos that might be talking about prerequisite subjects.
Can we directly encrypt kubernetes secret like using any encryption algorithm we encrypt the secret value and after that decrypt that value in pod and use it. So that anyone cannot access secret without private key. Does kubernetes or any other tool or methods will help to achieve this??
Awesome content as always. I have a silly question, can you seal on one cluster (kind) and deploy to another cluster (GKE) Or do I have to deploy to the cluster I used for the sealing
You have to deploy it inside the cluster where you sealed it. Those whole idea is that no one and nothing can unseal it with the access to the cluster that sealed it, and that includes sealed secrets controller.
True about git as the single source of truth, but how about modern CI/CD where there are multiple clusters to deploy to in the same pipeline (test, acceptance ,production)? Who is actually performing the kubeseal command? DevOps people usually do not have direct access to the (production) clusters, let alone should be able to create resources. How is key management working when all the secrets are spread across multiple git repos. How are clients reacting when their secret, although encrypted, is available to a wider audience in some git repo. How about migrating to a different cluster. Lots of questions, and I'm wondering if this solution is the way forward in any DevOps shop.
As a side note... Git is not the source of truth. The source of truth are "real" resources whereever they are. Git is where the desired state is kept and that desired state might or might not be the same as the actual state.
Hi Quick questions But i can take this manifest , and run against my local kubernetes , where kubeseal is setup and i can decrypt right ? How is this secret anymore? Am i missing anything :(
You can decrypt it because you have access with permissions to do that. The same is true for any other solution. For example, if you have access to Hashi Vault, you can decrypt secrets stored there. So, what you need to make sure that you setup k8s RBAC well and do not give admin access to everyone. Also, just in case that wasn't clear in the video, you can decrypt secrets running in the same Namespace of the same cluster. You cannot encrypt a secret in one cluster and then apply it and decrypt it in another. You cannot even move it to a different Namespace.
Great video!! Congratulations, what If we lost the cluster? No way to recover secrets anymore? So I think that after installing the controller you might need to make a backup of the cluster data.
The same is true for any secrets management solution. If wherever secrets are located is lost, secrets are lost as well and, in all those cases, the solution is to restore a backup. You can do that easily with Sealed Secrets.
Are you referring to the few clips before the animation/title? If you are, it would be hard to bring them back. Editing is now done by an agency and they would not be able to choose the best moments. Btw. I loved that part.
What is the best practice in a multi tenant cluster? When I seal the secret in a certain namespace. Can it be unsealed in any namespace because all namespaces are using the same Bitnami controller?
As the key is being generated during the installation of SealedSecret operator and kept on the given cluster, SealedSecrets are valid as long as we do not reinstall/redeploy the cluster. If we do - we would have another key and the SealedSecrets generated and stored in git won't work. So wondering how we can handle this scenario, where we need to have a solution from ground-up, so stand-up the cluster and deploy everything including secrets from git. If I'm not mistaken - this is not going to work?
Normally, redeployment (upgrade) of a cluster would not result in any issues as long as you are doing it using rolling updates. Now, if you have to move to a different cluster or you loose a cluster completely and need to create a new one, you can restore encryption keys from a backup. Please take a look at github.com/bitnami-labs/sealed-secrets#how-can-i-do-a-backup-of-my-sealedsecrets
@@DevOpsToolkit ah I was looking for this, casue in the vide it was not mentioned what would happen if the storage craps out or we had to redeploy the operator from scratch, there would be no way we could decrypt the secrets.
This will not work if you have several clusters (dev, staging, prod...) since on each cluster will have a different public/private key. You can either have a secret per environment which is not ideal or bring your own certificate and share it between clusters which is also not ideal since you will have to rotate it yourself.
Normally, you have common and environment-specific manifests or customizations. For example, canary manifests make sense in prod but not necessarily in other envs. Secrets can be in that same group.
Hi. My applications are deploy with Helm chart and ArgoCD. Before, I create all my secrets for all my environnement, and templated in my Delpoyment template which secrets use. But, there is a way to let Helm deploy the sealed secrets with ArgoCD without the need to apply the sealedscrets manually? For the records, I deploy 70 iterations of the same app with differents secrets (one for each client database). It could be cool if Helm could use the good sealed secrets file in git to create the secret needed for my deploy. Any idea how to do this in Helm with Argocd?
I don't know of a way to do that directly in Helm. That does not mean that there isn't a way, but, rather, that I haven't tried it. Normally, I would seal secrets as part of my pipelines/workflows which I use to modify Argo CD manifests and push to Git.
If it's fully destroyed (nothing left) and you need to create it from scratch (which rarely, if ever, happens, than you would need to create secrets from scratch as well.
I forgot to mention backups. If you do back up secrets, you can restore it or use it to retrieve secrets in case you want to put them somewhere else (e.g. a new cluster).
hi Victor, what do you say about Soluto/kamus. it has much more options. also gitops friendly and mentioned in many articles. also as i have seen permissions are more flexible . thanks
Thanks for explaining it n a so simple way. But, I think if someone manages to get access to a sealed secret, all they would need is a sealed secret controller to check the contents in the secret. I don't think it is really safe unless there is a key used for encryption and the same key is configured in the controller for decryption. This key I'm talking about, can be any random key.
The keys are in your Kubernetes cluster (the one used to encrypt the secret). That means that secrets are as safe (or unsafe) as your Kubernetes cluster is. That's where I believe GitOps plays a crucial role. If you are using something like Argo CS or Flux, no one needs to have access to Kubernetes clusters, thus making it relatively secure.
would be interesting to have a comparison between Sealed Secrets and SOPS. I found SOPS a bit more practical in the sense that you can use Cloud-managed key services (like KMS) to encrypt while ensuring IAM-based access to those keys. I haven't looked much into Sealed Secrets until now, but I wonder how it compares and whether Sealed Secrets can use KMS keys for example?
A while ago, when I was looking for a solution like that for my use-cases, I saw a huge difference in stars and forks between gopass and kubeseal and jumped straight into the latter. Still, the number of stars and forks does not mean that one is better than the other. Is it good? Should I reevaluate my options and give gopass a go?
@@joebowbeer SOPS is one of the most commonly used alternatives. Unlike SealedSEcrets, it is not k8s-only so it can be used for other purposes as well.
I was about to correct you on base64 encoding not being encryption until future you beat me to it :) good video
Your videos are awesome, very helpful, thank you🙂
Please consider skipping the intro music part, and start right away, like the content, subbed
Noted. No intro, or reduced intro starting from the next one :)
@@DevOpsToolkit thank you
you are an amazing teacher, thank you. Hello from Ukraine
I hope you're safe.
I love your explanation :D you got a subscriber. Thanks for the vid
subscribed. This is just what I was looking for, really well explained. Worked like a charm. Thank you and keep up with the awesome videos.
Thanks for the sub!
Thanks for explaining so beautifully, loved this video.
This reminds me a bit good old ansible-vault encrypt/decrypt
It's indeed similar.
not everyone here is a software engineer some folks wants to get into devops r cka soo next time when u choose a topiic like this plz make it as like for beginners to advanced level
That's a good suggestion. I'll do my best to either start earlier in a story or link to other videos that might be talking about prerequisite subjects.
Thanks ❤!!
awasome thanks
Awesome learning experience always.
Can we directly encrypt kubernetes secret like using any encryption algorithm we encrypt the secret value and after that decrypt that value in pod and use it. So that anyone cannot access secret without private key. Does kubernetes or any other tool or methods will help to achieve this??
SealedSecrets does that except that it decrypts it into a Kubernetes secret and not a Pod.
Awesome content as always.
I have a silly question, can you seal on one cluster (kind) and deploy to another cluster (GKE)
Or do I have to deploy to the cluster I used for the sealing
You have to deploy it inside the cluster where you sealed it. Those whole idea is that no one and nothing can unseal it with the access to the cluster that sealed it, and that includes sealed secrets controller.
True about git as the single source of truth, but how about modern CI/CD where there are multiple clusters to deploy to in the same pipeline (test, acceptance ,production)? Who is actually performing the kubeseal command? DevOps people usually do not have direct access to the (production) clusters, let alone should be able to create resources. How is key management working when all the secrets are spread across multiple git repos. How are clients reacting when their secret, although encrypted, is available to a wider audience in some git repo. How about migrating to a different cluster. Lots of questions, and I'm wondering if this solution is the way forward in any DevOps shop.
I changed my mind since that video and now i believe that the best option is to use external secrets operator (ESO).
As a side note... Git is not the source of truth. The source of truth are "real" resources whereever they are. Git is where the desired state is kept and that desired state might or might not be the same as the actual state.
Hi
Quick questions
But i can take this manifest , and run against my local kubernetes , where kubeseal is setup and i can decrypt right ?
How is this secret anymore?
Am i missing anything :(
You can decrypt it because you have access with permissions to do that. The same is true for any other solution. For example, if you have access to Hashi Vault, you can decrypt secrets stored there. So, what you need to make sure that you setup k8s RBAC well and do not give admin access to everyone. Also, just in case that wasn't clear in the video, you can decrypt secrets running in the same Namespace of the same cluster. You cannot encrypt a secret in one cluster and then apply it and decrypt it in another. You cannot even move it to a different Namespace.
We used to do this back in the Chef days 😉
Great video!! Congratulations, what If we lost the cluster? No way to recover secrets anymore? So I think that after installing the controller you might need to make a backup of the cluster data.
The same is true for any secrets management solution. If wherever secrets are located is lost, secrets are lost as well and, in all those cases, the solution is to restore a backup. You can do that easily with Sealed Secrets.
I miss these intros! Bring em back! :D
Are you referring to the few clips before the animation/title? If you are, it would be hard to bring them back. Editing is now done by an agency and they would not be able to choose the best moments.
Btw. I loved that part.
@@DevOpsToolkit yes I meant those. Too bad :(
Actually, I might try to bring it back. I think that, by now, the editor learned how DevOps and programming works :)
@@DevOpsToolkit That would be awesome! :)
What is the best practice in a multi tenant cluster? When I seal the secret in a certain namespace. Can it be unsealed in any namespace because all namespaces are using the same Bitnami controller?
No. A secret sealed in a Namespace is available only in that Namespace.
As the key is being generated during the installation of SealedSecret operator and kept on the given cluster, SealedSecrets are valid as long as we do not reinstall/redeploy the cluster.
If we do - we would have another key and the SealedSecrets generated and stored in git won't work.
So wondering how we can handle this scenario, where we need to have a solution from ground-up, so stand-up the cluster and deploy everything including secrets from git.
If I'm not mistaken - this is not going to work?
Normally, redeployment (upgrade) of a cluster would not result in any issues as long as you are doing it using rolling updates. Now, if you have to move to a different cluster or you loose a cluster completely and need to create a new one, you can restore encryption keys from a backup. Please take a look at github.com/bitnami-labs/sealed-secrets#how-can-i-do-a-backup-of-my-sealedsecrets
@@DevOpsToolkit ah I was looking for this, casue in the vide it was not mentioned what would happen if the storage craps out or we had to redeploy the operator from scratch, there would be no way we could decrypt the secrets.
@@vladoportos Yep. You need to create backups to be able to recuperate from destroyed (unrecuperable) cluster failure.
great video, thanks for that :)
Awesome thank you!
This will not work if you have several clusters (dev, staging, prod...) since on each cluster will have a different public/private key. You can either have a secret per environment which is not ideal or bring your own certificate and share it between clusters which is also not ideal since you will have to rotate it yourself.
Normally, you have common and environment-specific manifests or customizations. For example, canary manifests make sense in prod but not necessarily in other envs. Secrets can be in that same group.
Hi. My applications are deploy with Helm chart and ArgoCD. Before, I create all my secrets for all my environnement, and templated in my Delpoyment template which secrets use. But, there is a way to let Helm deploy the sealed secrets with ArgoCD without the need to apply the sealedscrets manually? For the records, I deploy 70 iterations of the same app with differents secrets (one for each client database). It could be cool if Helm could use the good sealed secrets file in git to create the secret needed for my deploy. Any idea how to do this in Helm with Argocd?
I don't know of a way to do that directly in Helm. That does not mean that there isn't a way, but, rather, that I haven't tried it.
Normally, I would seal secrets as part of my pipelines/workflows which I use to modify Argo CD manifests and push to Git.
What happens with secrets if cluster gets destroyed? How would one decrypt it if git was the only place it was kept?
If it's fully destroyed (nothing left) and you need to create it from scratch (which rarely, if ever, happens, than you would need to create secrets from scratch as well.
I forgot to mention backups. If you do back up secrets, you can restore it or use it to retrieve secrets in case you want to put them somewhere else (e.g. a new cluster).
Is this what you would put in your " The Best DevOps Tools in 2022" for secrets mgt if you'd had that category?
It would be one of the winners in the secrets category. It's not the most secure option, but it's good and convenient.
thanks man for share it
Any time
hi Victor, what do you say about Soluto/kamus. it has much more options. also gitops friendly and mentioned in many articles. also as i have seen permissions are more flexible . thanks
I did not yet use it. Let me try it out first...
It's called base64 decoding. Call it decoding only, not decryption.
You're right. I noticed that after I uploaded the video (and RUclips does not allow re-uploads).
Should I put sealed secrets in its own namespace? This way I can restrict access to it via namespace.
Yes. Sealed secrets should be namespaced. I never even tried to make them cluster-wide. I doubt that is even possible.
@@DevOpsToolkit Actually, you can 'scope' a sealed secret to be cluster-wide, namespace-wide or strict (default) with the --scope option.
Thanks for explaining it n a so simple way. But, I think if someone manages to get access to a sealed secret, all they would need is a sealed secret controller to check the contents in the secret. I don't think it is really safe unless there is a key used for encryption and the same key is configured in the controller for decryption. This key I'm talking about, can be any random key.
The keys are in your Kubernetes cluster (the one used to encrypt the secret). That means that secrets are as safe (or unsafe) as your Kubernetes cluster is. That's where I believe GitOps plays a crucial role. If you are using something like Argo CS or Flux, no one needs to have access to Kubernetes clusters, thus making it relatively secure.
would be interesting to have a comparison between Sealed Secrets and SOPS. I found SOPS a bit more practical in the sense that you can use Cloud-managed key services (like KMS) to encrypt while ensuring IAM-based access to those keys. I haven't looked much into Sealed Secrets until now, but I wonder how it compares and whether Sealed Secrets can use KMS keys for example?
Adding the comparison to my TODO list... :)
How do you update existing secret that is sealed?
You just create a new manifest or update the existing manifest and apply it.
@@DevOpsToolkit thank you for your response and this amazing video 😃
One hint: gopass also supports Git-based secrets encryption with Kubernetes integration, among others: github.com/gopasspw/kubectl-gopass
A while ago, when I was looking for a solution like that for my use-cases, I saw a huge difference in stars and forks between gopass and kubeseal and jumped straight into the latter. Still, the number of stars and forks does not mean that one is better than the other.
Is it good? Should I reevaluate my options and give gopass a go?
Mozilla SOPS is an alternative? Flux v2 has built-in SOPS support.
@@joebowbeer SOPS is one of the most commonly used alternatives. Unlike SealedSEcrets, it is not k8s-only so it can be used for other purposes as well.
The company executives not understanding GitOps secrets management like this are the ones who should be fired. Stupid is everywhere.
Not understanding something is not a problem. Not listening to those who do understand something is the real issue.
@@DevOpsToolkit Your videos are excellent and a huge saver. Thank you