Never stop making these vids. More DevOps "professionals" should watch your videos cause from experience they're severely lacking and should watch your videos every day
Simply awesome, so glad I found your videos, they are absolutely straightforward, to the point and very intuitively presented. I did all the steps as suggested without any hitch and so happy to see our vault up running with all the secrets injected. Now trying to write a Spring Boot client that could connect to vault and fetch the secret. Thanks a lot Marcel, very much appreciated.
Hey mate. I love your short and crisp content on everything that you teach us. I would however would like to request you to do a series on "service mesh" like Consul\LinkerD\Istio whatever that suits you but Consul certainly would be a favourable choice considering it being a Hashicorp product line and how seamlessly it integrates with Vault and other components. Keep doing the great work for the community!!!
cool video 😌 but @Devops Guy, suggest if we provision a provisioner of vault in namespace abc microservices and want same secret in another microservice namespace xyz then do we have to provision another provisioner of vault in xyz namespace as well?
Question: I was under the impression that the Vault implementation would be transparent to the application. I would appreciate seeing how an existing app using a Kubernetes secret is implemented demonstrating the transparency. Thank you. GREAT VIDEOS!
A Kubernetes secret is just an abstraction that mounts a secret file to disk where the application can read it. Vault has a similar abstraction that uses a sidecar injector to inject a secret from Vault into disk where the application can read it. The application does not have to be Vault aware. Here is an example of an app that uses a database credential on file with Vault : ruclips.net/video/IWCOptiCKqI/видео.html
Nice video. One thing i really dont understand is that If we opt for Hashicorp vault or even Azure Key vault with Kubernetes, in both cases a volume is mounted with clear text password inside the pod. Which to me is a security compromise. We introduce additional complexity in our deployments to implement these vaults but then the password is still in clear text. Not saying that these vaults dont improve the security in general but why then to have a password in clear text?
which is better or which is recommended , to have Vault as pod in Kubernetes or to have it installed as a cluster on separte VMs and connect it to kubernetes ?
Good question, there is no one size fits all answer to this one. Hashicorp recommends Vault be isolated on its own cluster and joined with the cluster where your services are on.
One small doubt. Why does the injector need a service account that has access to the mutating webhooks. The deployment that uses the service account itself is responsible for exposing the endpoint that is used by the mutating web hook. In that case, why do we need the deployment to give the permissions to get list and watch mutating webhooks.
Awesome! Could you explain dynamic way using google cloud with KMS? Also I'm looking forward to see how it auto unseal automatically using GCP services. Thank you so much for your time!
Thank you! I would start with the HashiCorp learning walkthroughs, they're great to follow and there is a KMS unseal one here learn.hashicorp.com/vault/operations/autounseal-gcp-kms
Hey Nice video but i was looking for how to inject those secrets as environment variables which should be present permanently as it happened with the secret objects in kubernetes. I tried benzai cloud also but it is not working.
This is probably one of the most popular questions. At the time of this video I don't think it was possible. I'd recommend to checkout the vault docs, also checkout their GitHub repo and search through historical issues as I am sure they have answered it over there 💪🏽
Thank you very much for Your tutorials, everything is very clear. I have one question please. Regarding secret injection to application, everything is clear, but how works application authentication to Vault in Kubernetes, for example, when my application need manually go to Vault and get or create secrets periodically. For example if skip Kubernetes for a moment, I create policy, then create token for than policy in Vault. Now I can use this token for authentication during http API call to Vault, to get or create secrets by application. But these tokens has limited lifetime. How this can be achieved in Kubernetes? Or when we use Vault Agent for injecting secrets, application also already has opportunity to get and create secrets according to its policy permissions by http API without authentication by token? Please explain a little if You will have time or may be give some useful links, where I can read about this? Thank you. 🙏
This is a very interesting question. Personally have not dealt with it, but I have come across the vault agent which has new capabilities of auto authenticating, and refreshing tokens which might help. It seems like its a daemon that can take care of rotating tokens to Vault on a server and your application can use that token periodically to make calls to Vault. This will take the burden away from your app having to refresh tokens. There is an interesting video here with a timestamp where HashiCorp discusses it's features. Hope it helps ruclips.net/video/zDnIqSB4tyA/видео.html
I've been using vault directly, as in my microservices make requests directly to vault. I'm curious how you handle rolling secrets with TTL with injection?
I've not worked on a microservice that uses rotating secrets, but I would imagine you'd need some form of middleware in the code that watches a file for changes and hot reloads it when a change is detected. That would allow you to consume rotating secrets. I'd also ensure the TTL overlaps slightly so you dont have windows where the key is expired. I've made a video on dynamic secret generation if that interests you ruclips.net/video/IWCOptiCKqI/видео.html
@@MarcelDempers Yea, watching the file makes sense. At that point though you may as well just reach out to the API though. Either way the consuming code is going to need to refresh their connections anyhow. If you aren't pulling in the lease then you'll have to hardcode the refresh rate. I randomize the refresh between 40-60% of the lease so all consumers aren't going off at once.
I would be hesitant to couple all my microservices to the vault API. That means services would rely on Vault being up and also would add code to each service to understand how to talk to vault. Also, every service would need to authenticate with Vault which makes it more concerning especially when you scale to 100s of services. Having the service watch a file, would avoid all that coupling and authentication concern and leave it up to vault and its side cars to take care of secret lifecycle.
I like these types of discussions and I'm sure the community would benefit from your thoughts , feel free to join our discord here. There is a separate channel for security: marceldempers.dev/community
@@MarcelDempers Oh I'm definitely hesitant. I'm with you, in that tightly of coupling is concerning. However my use-case requires a database per customer. I could still go with a filewatcher but that could potentially run into the same issues as above, with less potential warning (ie, Vault is down and thus not injecting secrets and the credentials expire).
Hello. The video was really nice, but i was not able to get it working unfortunately :( i did follow the steps but i didn't see the initial pods to be running, or even an error and the worst is that even the secrets are not displayed there. Anybody have any idea what could have went wrong?
I would highly recommend checking out part 1 and 2 for more in depth steps on getting Vault up and running and initialized correctly. The vault needs to be fully initialized, unsealed and you need to ensure you can make API calls to it. This video has a lot of detail around the secret injection but it sounds like you need take a few steps back and look more at standing up the vault properly. Here you go 💪🏽 #1 ruclips.net/video/L_o_CG_AGKA/видео.html #2 ruclips.net/video/Eapf-OWbKH8/видео.html
This was very helpful! I'm trying to understand how we can have the secret password from vault injected as environment variable when we initialize the pod. I'm installing Sonarqube using helm on my k8s cluster, the postgres DB credential is getting created as an environment variable with the chart, how can I inject the secret to environment variable. Here was a post of mine on stackoverflow: stackoverflow.com/questions/61239479/injecting-vault-secrets-into-kubernetes-pod-environment-variable Helm chart for sonarqube I'm using : github.com/Oteemo/charts/tree/master/charts/sonarqube
Thank you! After bit of reading i believe Vault does not yet support ENV var secret injection. This may require some hacking like reading a secret off disk from an init container where you run some custom code that potentially creates the ENV vars before your main application starts up. Not sure if it would work but seems like an option. github.com/hashicorp/vault-k8s/issues/14
Init:0/1 problem λ kubectl get all -n vault-example NAME READY STATUS RESTARTS pod/basic-secret-6699bfc678-2vlqj 0/2 Init:0/1 0 pod/vault-example-0 1/1 Running 1 pod/vault-example-agent-injector-86d756fb54-phx6n 1/1 Running 0
Never stop making these vids. More DevOps "professionals" should watch your videos cause from experience they're severely lacking and should watch your videos every day
What a series of Hashicorp Vault. Really enjoyed every video, really appreciated!
I've found you by accident. Dope way of passing knowledge. Subscribed ✅
Simply awesome, so glad I found your videos, they are absolutely straightforward, to the point and very intuitively presented. I did all the steps as suggested without any hitch and so happy to see our vault up running with all the secrets injected. Now trying to write a Spring Boot client that could connect to vault and fetch the secret. Thanks a lot Marcel, very much appreciated.
I love the way you teaching👏👏..
your computer setup is dope 🎁
It was short and crisp, looking forward to a new video for dynamic password creation in vault
Pretty cool you're giving all these high quality videos out. Much respect bro.
Hey mate. I love your short and crisp content on everything that you teach us. I would however would like to request you to do a series on "service mesh" like Consul\LinkerD\Istio whatever that suits you but Consul certainly would be a favourable choice considering it being a Hashicorp product line and how seamlessly it integrates with Vault and other components. Keep doing the great work for the community!!!
Excellent. One of the best videos on secret injection.
you are awesome dude.. please never stop teaching!.. love your style 😎😎😎😎
loaded with a lot of knowledge Thanks Man!
This is just great! You speed up my work a lot!
I like your videos, informative and straight forward.
Very nice explanation! Thank you
You are awesome man! Excellent explanations.
so what is the role mutating admission webhook in this videos, do we have to create this one separately.
Nice coffee machine 😀
Aww!! this is awesome...... thanks a lot for this beautiful content. looking forward for next video :)
DevOps Guy, is there any chance to not have those secrets injected as a file but as an environment variable? Thanks man.
Really nice and useful video a big thank
cool video 😌 but @Devops Guy, suggest if we provision a provisioner of vault in namespace abc microservices and want same secret in another microservice namespace xyz then do we have to provision another provisioner of vault in xyz namespace as well?
what would be the way to do this if the Vault exists outside of the k8s cluster?
Very helpful as usual!
Could this same setup tweaked to create kubernetes secret option rather than mounting as volume?
Well done. Cool channel with useful content
Question: I was under the impression that the Vault implementation would be transparent to the application. I would appreciate seeing how an existing app using a Kubernetes secret is implemented demonstrating the transparency. Thank you. GREAT VIDEOS!
A Kubernetes secret is just an abstraction that mounts a secret file to disk where the application can read it.
Vault has a similar abstraction that uses a sidecar injector to inject a secret from Vault into disk where the application can read it.
The application does not have to be Vault aware.
Here is an example of an app that uses a database credential on file with Vault : ruclips.net/video/IWCOptiCKqI/видео.html
thanks a lot! perfect explanation
Nice video. One thing i really dont understand is that If we opt for Hashicorp vault or even Azure Key vault with Kubernetes, in both cases a volume is mounted with clear text password inside the pod. Which to me is a security compromise. We introduce additional complexity in our deployments to implement these vaults but then the password is still in clear text. Not saying that these vaults dont improve the security in general but why then to have a password in clear text?
what do you mean it is clear text, within the pod? You should have RBAC in place so that not everyone can exec into the pod.
which is better or which is recommended , to have Vault as pod in Kubernetes or to have it installed as a cluster on separte VMs and connect it to kubernetes ?
Good question, there is no one size fits all answer to this one.
Hashicorp recommends Vault be isolated on its own cluster and joined with the cluster where your services are on.
@@MarcelDempers thank you for your answer . got it
could pls let us know how to access those secrets through environment variables
Very useful man! Thanks
How to inject secrets as env variables all at a same time using this method?
Could you please show a demo how to retrieve the secrets from vault using kubernetes config map. It would be of great help!
One small doubt. Why does the injector need a service account that has access to the mutating webhooks. The deployment that uses the service account itself is responsible for exposing the endpoint that is used by the mutating web hook. In that case, why do we need the deployment to give the permissions to get list and watch mutating webhooks.
please make a video on "argocd-secret-management-with-argocd-vault-plugin"
Hi, do you know how can i put this file into environment variable inside the pod? I'm search a lot but i can't find an answer, thanks!
Awesome! Could you explain dynamic way using google cloud with KMS? Also I'm looking forward to see how it auto unseal automatically using GCP services. Thank you so much for your time!
Thank you! I would start with the HashiCorp learning walkthroughs, they're great to follow and there is a KMS unseal one here learn.hashicorp.com/vault/operations/autounseal-gcp-kms
Not a huge fan of the purple VSCode background lol, but otherwise I love the content, keep up the great work!
I'm keen for a change, got any ideas for cool ones ?
@@MarcelDempers Not really lol, found one on vscodethemes.com, I believe it was Winter is coming theme
Hey Nice video but i was looking for how to inject those secrets as environment variables which should be present permanently as it happened with the secret objects in kubernetes. I tried benzai cloud also but it is not working.
This is probably one of the most popular questions. At the time of this video I don't think it was possible. I'd recommend to checkout the vault docs, also checkout their GitHub repo and search through historical issues as I am sure they have answered it over there 💪🏽
@@MarcelDempers Thanks for the reply, Yes I have tried multiple things but no luck🙂
Thank you very much for Your tutorials, everything is very clear. I have one question please. Regarding secret injection to application, everything is clear, but how works application authentication to Vault in Kubernetes, for example, when my application need manually go to Vault and get or create secrets periodically. For example if skip Kubernetes for a moment, I create policy, then create token for than policy in Vault. Now I can use this token for authentication during http API call to Vault, to get or create secrets by application. But these tokens has limited lifetime. How this can be achieved in Kubernetes? Or when we use Vault Agent for injecting secrets, application also already has opportunity to get and create secrets according to its policy permissions by http API without authentication by token? Please explain a little if You will have time or may be give some useful links, where I can read about this? Thank you. 🙏
This is a very interesting question. Personally have not dealt with it, but I have come across the vault agent which has new capabilities of auto authenticating, and refreshing tokens which might help. It seems like its a daemon that can take care of rotating tokens to Vault on a server and your application can use that token periodically to make calls to Vault. This will take the burden away from your app having to refresh tokens. There is an interesting video here with a timestamp where HashiCorp discusses it's features. Hope it helps ruclips.net/video/zDnIqSB4tyA/видео.html
@@MarcelDempers Thank you very much, that was very helpful.
Very helpful
Any way to monitor it?
I've been using vault directly, as in my microservices make requests directly to vault. I'm curious how you handle rolling secrets with TTL with injection?
I've not worked on a microservice that uses rotating secrets, but I would imagine you'd need some form of middleware in the code that watches a file for changes and hot reloads it when a change is detected. That would allow you to consume rotating secrets. I'd also ensure the TTL overlaps slightly so you dont have windows where the key is expired. I've made a video on dynamic secret generation if that interests you ruclips.net/video/IWCOptiCKqI/видео.html
@@MarcelDempers Yea, watching the file makes sense. At that point though you may as well just reach out to the API though. Either way the consuming code is going to need to refresh their connections anyhow. If you aren't pulling in the lease then you'll have to hardcode the refresh rate.
I randomize the refresh between 40-60% of the lease so all consumers aren't going off at once.
I would be hesitant to couple all my microservices to the vault API. That means services would rely on Vault being up and also would add code to each service to understand how to talk to vault. Also, every service would need to authenticate with Vault which makes it more concerning especially when you scale to 100s of services. Having the service watch a file, would avoid all that coupling and authentication concern and leave it up to vault and its side cars to take care of secret lifecycle.
I like these types of discussions and I'm sure the community would benefit from your thoughts , feel free to join our discord here. There is a separate channel for security: marceldempers.dev/community
@@MarcelDempers Oh I'm definitely hesitant. I'm with you, in that tightly of coupling is concerning. However my use-case requires a database per customer. I could still go with a filewatcher but that could potentially run into the same issues as above, with less potential warning (ie, Vault is down and thus not injecting secrets and the credentials expire).
Hello. The video was really nice, but i was not able to get it working unfortunately :( i did follow the steps but i didn't see the initial pods to be running, or even an error and the worst is that even the secrets are not displayed there. Anybody have any idea what could have went wrong?
I would highly recommend checking out part 1 and 2 for more in depth steps on getting Vault up and running and initialized correctly. The vault needs to be fully initialized, unsealed and you need to ensure you can make API calls to it. This video has a lot of detail around the secret injection but it sounds like you need take a few steps back and look more at standing up the vault properly. Here you go 💪🏽
#1 ruclips.net/video/L_o_CG_AGKA/видео.html
#2 ruclips.net/video/Eapf-OWbKH8/видео.html
👌
mirin the buff
This was very helpful! I'm trying to understand how we can have the secret password from vault injected as environment variable when we initialize the pod. I'm installing Sonarqube using helm on my k8s cluster, the postgres DB credential is getting created as an environment variable with the chart, how can I inject the secret to environment variable.
Here was a post of mine on stackoverflow: stackoverflow.com/questions/61239479/injecting-vault-secrets-into-kubernetes-pod-environment-variable
Helm chart for sonarqube I'm using : github.com/Oteemo/charts/tree/master/charts/sonarqube
Thank you! After bit of reading i believe Vault does not yet support ENV var secret injection. This may require some hacking like reading a secret off disk from an init container where you run some custom code that potentially creates the ENV vars before your main application starts up. Not sure if it would work but seems like an option.
github.com/hashicorp/vault-k8s/issues/14
The only part I am worried is "how simple and easy to do". BUT you have no clue 😐
Init:0/1 problem
λ kubectl get all -n vault-example
NAME READY STATUS RESTARTS
pod/basic-secret-6699bfc678-2vlqj 0/2 Init:0/1 0
pod/vault-example-0 1/1 Running 1
pod/vault-example-agent-injector-86d756fb54-phx6n 1/1 Running 0