You're an excellent teacher Marcel, thank you for making these videos. Your vlog format is groundbreaking in tech. While there are plenty of others on youtube offering fantastic content, you've introduced (at least to me) an element that has made it so much easier to take the information in. Well done man!
Man this is just awesome, I'm beating myself for not understanding, after watching this I can assure myself I got the idea I remember this quote, You can't connect the dots forward - only backward - Steve Jobs Thank you for the lecture marcel, You made my day.
Bro out here making every DevOps engineers life so much easier, thanks the school system around the globe should take you as an example on how to transfer knowledge....
Hi Marcel Thank you so much for this simple explanation with a REAL WORLD example of persistent volumes. I have been wracking my brains trying to tie together pv, pvc, and statefulsets. You explained and demonstrated in 20 minutes what other people couldn't get across in hours of demos. Love your channel, keep it up mate 👌👍🙂
Marcel, your stateful and persistance videos demos are brilliant, thank you for putting in the work to share your knowledge, your delivery is brilliant
Good video generally, but there's a misconception about containers. A container is not a process, it's a sandboxed runtime environment in which you can run one or more processes. You can stop a container (`docker stop `) which terminate all processes inside of a container and then run it again (`docker start `) and the files will still be there. Kubernetes never bothers to restart a container - when it restarts a pod it recreates containers with the same IP address assigned, etc. so you might think that containers are completely stateless (and they're typically treated as such) but a container keeps its filesystem state after all of its processes terminate which is pretty useful when it comes to retrieving logs and inspecting what caused the termination.
Marcel you're a good man. You always explain things very clearly and dough you type as fast as the light you repo in GitHub shows everything. So many many thanks. I was stuck trying to bind a local PV to a mongo db StatefulSet , looking at your code(dough for Postgres ) made me find a way to debug my pv.yaml.. it results that was a typo in the nodeAffinity key.. thank you so much again. Vinny
Thank you so much for this. I’ve been in class all week and studying my ass off and just have NOT understood the pv claim until now. Now I can get some sleep. Lol
Thanks alot Marcel, I finally understood the concepts of PV, PVC and how to use them with Postgres thanks to you. Massive thanks! Also I would like to request if you could teach how to do authentication and authorizations from nginx-ingress using Keycloak, as I am sure many people struggle with it... And there arent any RUclipsrs like you, who teach from the basics and build up concepts upon that, rather they usually just assume that we just know stuff. Also, since KeyCloak is really popular Auth as a Service, so I am sure it would be very helpful for others too, so please consider doing a video on it, please. Thanks a lot in advance.
Great Video, breaking down the content in ingestible bytes. One improvement I'd suggest is timestamping your video so that people can skip ahead to the exact part they are looking for. Keep doing you man your content is great
Thanks a lot, Marcel this is the best video about the concept that I've found on RUclips, your explanation coming from docker volumes creates a solid ground to easily understand the kubernetes part :smile:
Hi Gaurav, is this end of K8s series, please add if few more topics pending. and i must saym your tutorial made my concep clear and handson experianced.
Hey Marcel, love the videos! I think there is a small mistake with your explanation on PVCs claiming storage from PVs. You cannot have several PVCs bound to one PV, once a PVC is bound to a PV the PV will not be able to bind to another PVC but many pods can use the same PVC. You create PVs to configure a certain way to claim storage and to use it you bind a PVC to it. If you aren't using PVs the storage class dynamically creates and provisions a PV for the requested PVC. Keep up the great content!
Thanks! I have a single tennet app that will have multiple copies of the same pod running in different namespaces with different persistant volume claims. Many of these resources have the same name, but are just deployed in different namespaces. It would be great to see a video about how a single pod .yaml could be used in that way to create different volumes.
Thank you, the guide was very clear and concise. Do you have a guide on how to handle a "real world" project in the different phases? For example I imagine one would develop in local on minikube, then deploy on the cloud (AWS, GCP, etc), maybe having TEST/PRODUCTION as different clusters. How do you manage all this? Does it make sense to have in the same git repo different kubernetes folders, one for each environment, containing the appropriate yml? Or would you rather have different branches? A guide on this would be great.
Hi Marcel and thanks you for the great job you do! Can you give a hint or direction please. I've deployed wp with pvc using nfs-provisioner on bare-metal. nfs share is located on the neighbouring vm host in the same subnet and basically both k8s nodes and nfs node use the same physical disk. What I see is that i can easily store anything on this share manually from any pod but WP + mysql is very slow when using this nfs (e.g. it take up to 15 sec to read a few KB files). perms on the share are 777. Access mode readwritemany if it makes any difference. I've googled for this issues and it turns out quite common but there is no specific solution or reason that I could find. Any advices please?
How does it work, then, if I have multiple apps that need to share the same physical storage. From what it sounds, I still need to create a pv for all of them, but can mount them to the same physical path in the storage class, nfs for example. But then how does the storage amount come into play? How can a pv reliably tell a claim “hey, I have this much storage available, use me.” If there is overlap in the physical storage?
Hi, i got a qusetion, if my volume is stored application's log file, this volume should be oversized, how could i logrotate in this volume to manage the log file size? do u have any suggestion? thank u so much
I would recommend looking into Fluentd for shipping logs and not store logs in a volume. K8s has pod logs go to a container runtime folder and Fluentd mounts that directory to forward logs out. The container runtime is designed to rotate logs automatically. ruclips.net/video/6kmHvXdAzIM/видео.html
What is the best way to setup storage server and its configuration for kubernetes cluster persistence volume for production system? NFS or ISCASI any hardware brand specific. Thanks !
Hi Marcel, I've created a PersistentVolume using the hostpath StorageClassName and I mounted it in a mysql pod. The Persistent Volume hostpath is "/var/folders/data", and it's mounted in "/var/lib/mysql" in the Container. What's strange is that when I list the PV hostpath "/var/folders/data" it's empty. But when I create a test database and then delete and recreate the deployment the test database is still there. You know where the container "/var/lib/mysql" files could be found? I was expecting to find them in the PV hostpath "/var/folders/data". I'm using docker for mac.
Great video Marcel! However, I do have a question. I am able to persist data through the lifetime of the container / pod, but not able to do so if I restart my cluster. How do I achieve this? I am planning to use local storage class
For logging, I have a separate video in the pipeline, You would rather export logs from the app and not store it in a volume. Take a look at the EFK\ELK concepts, where basically every pod in K8s writes logs to stdout. It is best practise not to write logs to file and rather use stdout\stderr. These logs will automatically end up in the host docker logs which you can push out of the system using Fluentd\FluentBit. This allows you to centralise logs for all deployments\statefulsets running in your cluster under something like Elastic Search or 3rd party application.
Hi Thanks you for great video.! I did the same and tried to deploy jenkins but when i create pv and pvc . mnt/data is empty. and i used volumemount to var/jenkins_home.
but If we have a three node cluster one is master other two are worker first time our Postgres pod is schedule on worker1 now its PV on worker1 and second time after deleting the pod k8s schedule it on Worker2 now pod create its PV on Worker2 but now data is not persistence. because data on worker1 can not allocate to worker2 pod? What's a solution?
This is because you are using a deployment with PVs which is not meant for stateful workloads. You'll need to learn about StatefulSets ruclips.net/video/zj6r_EEhv6s/видео.html
Hey Marcel, great videos. After I altered the ConfigMap to use a different POSTGRES_USER and deleted/recreated everything including the PV, the old database user & table existed. Can you explain why that happened even after I deleted the PV? Is there more to "scrub" here? I assumed recreating everything would result in a fresh database with the new user credentials. Thanks!
This highly depends on the storage class you use. In this demo I use host path which means data is persisted on the node at a path (which can be changed). The get a fresh storage, you either have to change the path to get a new folder on the host. Remember your old data would remain. Or SSH to the node, clean it up, Or run another container with a host mount and carefully run "rm" command to delete the content
@@MarcelDempers Thank you for such a quick response! I used hostPath/your tutorial code too. If you have any advice on ssh-ing to the docker-desktop (Mac) node to clean up, I'm all ears. Thanks again for these videos, very helpful.
Hi Marcel, my setup is a bit unique, I have created a K8S cluster on a few VMs in our own private cloud and all the nodes are mounted with a single NFS drive (say /data), can I define a storageclass of hostpath type using /data? Does PV has to be mounted to all the nodes?
That depends on the app you're planning to run and how it uses the storage. If you plan on using a statefulset its recommended that each pod has its own storage. StatefulSet will use a storage class to provision a PV per pod and not share the same storage. Hostpath will work, but if you mount a shared storage to all the nodes, they may override each other's data. So unless you mount unique storage to each node, it may not be ideal. I'm new to NFS, but i would see if its possible to provision NFS dynamically using a storage class and see if the statefulset can issue PV + PVCs per pods using NFS dynamic provisioning. That way each pod will get its own NFS storage provisioned by k8s when you scale up and down
Checkout NFS provisioning using Rook rook.io/docs/rook/v1.4/nfs.html Seems interesting. If you have not worked with Statefulsets I'd highly recommend to ruclips.net/video/zj6r_EEhv6s/видео.html
Understanding StatefulSets for beginners👉🏽 ruclips.net/video/zj6r_EEhv6s/видео.html
You're an excellent teacher Marcel, thank you for making these videos. Your vlog format is groundbreaking in tech. While there are plenty of others on youtube offering fantastic content, you've introduced (at least to me) an element that has made it so much easier to take the information in. Well done man!
You teach in a clear and easy to unsertand manner, really important to learn about storage classes.
Man this is just awesome, I'm beating myself for not understanding, after watching this I can assure myself I got the idea
I remember this quote,
You can't connect the dots forward - only backward
- Steve Jobs
Thank you for the lecture marcel, You made my day.
Bro out here making every DevOps engineers life so much easier, thanks the school system around the globe should take you as an example on how to transfer knowledge....
Hi Marcel
Thank you so much for this simple explanation with a REAL WORLD example of persistent volumes. I have been wracking my brains trying to tie together pv, pvc, and statefulsets. You explained and demonstrated in 20 minutes what other people couldn't get across in hours of demos. Love your channel, keep it up mate 👌👍🙂
Possibly the best tutorial on the topic!!
Marcel, your stateful and persistance videos demos are brilliant, thank you for putting in the work to share your knowledge, your delivery is brilliant
Thanks Marcel! Would really love to see one on Argo Workflows/Events for setting up and triggering pipelines in kubernetes!
Good video generally, but there's a misconception about containers. A container is not a process, it's a sandboxed runtime environment in which you can run one or more processes. You can stop a container (`docker stop `) which terminate all processes inside of a container and then run it again (`docker start `) and the files will still be there. Kubernetes never bothers to restart a container - when it restarts a pod it recreates containers with the same IP address assigned, etc. so you might think that containers are completely stateless (and they're typically treated as such) but a container keeps its filesystem state after all of its processes terminate which is pretty useful when it comes to retrieving logs and inspecting what caused the termination.
You're on another level mate, so concise, so clever, so well spoken, love your videos
Great explaination!! After watching at least 10 other tutorials, yours was the first one that really worked!
This video is almost as good as you triceps. Respect
You are too good at this, your videos have helped me understand and breakdown those heavy jargons on K8s doc pages. Thank you.
Marcel you're a good man. You always explain things very clearly and dough you type as fast as the light you repo in GitHub shows everything. So many many thanks. I was stuck trying to bind a local PV to a mongo db StatefulSet , looking at your code(dough for Postgres ) made me find a way to debug my pv.yaml.. it results that was a typo in the nodeAffinity key.. thank you so much again. Vinny
Thank you so much for this. I’ve been in class all week and studying my ass off and just have NOT understood the pv claim until now. Now I can get some sleep. Lol
Thank you for this video, please make a video on the production-ready Postgres database.
You sir are amazing! Love your style and your editing and your teaching ability. You're like the Kubernetes god! Keep up the great work!
Thanks alot Marcel, I finally understood the concepts of PV, PVC and how to use them with Postgres thanks to you. Massive thanks!
Also I would like to request if you could teach how to do authentication and authorizations from nginx-ingress using Keycloak, as I am sure many people struggle with it... And there arent any RUclipsrs like you, who teach from the basics and build up concepts upon that, rather they usually just assume that we just know stuff. Also, since KeyCloak is really popular Auth as a Service, so I am sure it would be very helpful for others too, so please consider doing a video on it, please. Thanks a lot in advance.
this devops guy LIFTS!!
💪🏽
Suffering for the past week figuring out how to persist data on a database container has finalldy led me here to find the solution...
Great Video, breaking down the content in ingestible bytes. One improvement I'd suggest is timestamping your video so that people can skip ahead to the exact part they are looking for. Keep doing you man your content is great
excellent and well detailed! thank you for the explanation dude!
Every sec has information in your videos.
Thanks a lot, Marcel this is the best video about the concept that I've found on RUclips, your explanation coming from docker volumes creates a solid ground to easily understand the kubernetes part :smile:
You are really good at explaining things. Can you do a tutorial on tricep extensions? Thanks!
super underrated comment lol
Thanks for this amazing video, I was really struggling with this part in Kubernetes. It helped me a lot :)
Thanks for sharing. Gives me better understading than k8s doc.
Just Amazing ! Thank you very much man ! Keep Going !
Thanks for the great content Marcel
Hi Gaurav, is this end of K8s series, please add if few more topics pending.
and i must saym your tutorial made my concep clear and handson experianced.
Very informative
Great explaination!
Hey Marcel, love the videos! I think there is a small mistake with your explanation on PVCs claiming storage from PVs. You cannot have several PVCs bound to one PV, once a PVC is bound to a PV the PV will not be able to bind to another PVC but many pods can use the same PVC. You create PVs to configure a certain way to claim storage and to use it you bind a PVC to it. If you aren't using PVs the storage class dynamically creates and provisions a PV for the requested PVC. Keep up the great content!
Excellent video It helped me to clear my concept Thanks.
Thanks! I have a single tennet app that will have multiple copies of the same pod running in different namespaces with different persistant volume claims. Many of these resources have the same name, but are just deployed in different namespaces. It would be great to see a video about how a single pod .yaml could be used in that way to create different volumes.
Best in class 🙌✌️
Your channel is a godsend 🙏🙏🙏
very good video, tks Marcel.
Great video!
Thank you, the guide was very clear and concise.
Do you have a guide on how to handle a "real world" project in the different phases? For example I imagine one would develop in local on minikube, then deploy on the cloud (AWS, GCP, etc), maybe having TEST/PRODUCTION as different clusters. How do you manage all this? Does it make sense to have in the same git repo different kubernetes folders, one for each environment, containing the appropriate yml?
Or would you rather have different branches? A guide on this would be great.
Thank you kindly for this video. One question though. Why would a pvc specifies a storage class? isn't it already defined in a pv?
Greate, thanks a lot for this video it cant be better
awesome !!
great video thank you
Thank you so much
That Brilliant Guy who is Awesomeeeeee.....
Thank you
how would I add nodeselectos and affinity rules for statefulsets?
thanks a lot , very informative and useful
@13:40, How is the PersistentVolumeClaim referencing the PeristentVolume (example-volume) ? How does the pvc know which PV to use?
Thanks brother
Hi Marcel and thanks you for the great job you do!
Can you give a hint or direction please. I've deployed wp with pvc using nfs-provisioner on bare-metal. nfs share is located on the neighbouring vm host in the same subnet and basically both k8s nodes and nfs node use the same physical disk. What I see is that i can easily store anything on this share manually from any pod but WP + mysql is very slow when using this nfs (e.g. it take up to 15 sec to read a few KB files). perms on the share are 777. Access mode readwritemany if it makes any difference. I've googled for this issues and it turns out quite common but there is no specific solution or reason that I could find. Any advices please?
HERO!
Please a video on storage class for aws
How does it work, then, if I have multiple apps that need to share the same physical storage. From what it sounds, I still need to create a pv for all of them, but can mount them to the same physical path in the storage class, nfs for example.
But then how does the storage amount come into play? How can a pv reliably tell a claim “hey, I have this much storage available, use me.” If there is overlap in the physical storage?
Hi, i got a qusetion, if my volume is stored application's log file, this volume should be oversized, how could i logrotate in this volume to manage the log file size? do u have any suggestion? thank u so much
I would recommend looking into Fluentd for shipping logs and not store logs in a volume.
K8s has pod logs go to a container runtime folder and Fluentd mounts that directory to forward logs out.
The container runtime is designed to rotate logs automatically.
ruclips.net/video/6kmHvXdAzIM/видео.html
@@MarcelDempers thank u so much! its useful to a kubernetes beginner user.
What is the best way to setup storage server and its configuration for kubernetes cluster persistence volume for production system? NFS or ISCASI any hardware brand specific. Thanks !
Perfect thanks! Super .......
You're great. Good job!
The local file system path you specified, you would see files in there after creating the table right?
Hi Marcel, I've created a PersistentVolume using the hostpath StorageClassName and I mounted it in a mysql pod. The Persistent Volume hostpath is "/var/folders/data", and it's mounted in "/var/lib/mysql" in the Container. What's strange is that when I list the PV hostpath "/var/folders/data" it's empty. But when I create a test database and then delete and recreate the deployment the test database is still there. You know where the container "/var/lib/mysql" files could be found? I was expecting to find them in the PV hostpath "/var/folders/data". I'm using docker for mac.
Excellent
where is /var/lib/postgresql/data located? Is this location inside pod?
Great video Marcel!
However, I do have a question.
I am able to persist data through the lifetime of the container / pod, but not able to do so if I restart my cluster. How do I achieve this?
I am planning to use local storage class
how to backup and restore database in this case ?
Thx Great Video 👍👍👍👍
The tutorial doesn't work on my end. Keeps getting /var/lib/postgresql/data invalid permission.
Im running docker with k8s on windows 11
Hello. I have a question. What if you would want to save and mount the logs of a server application? Should you still use deployment or statefulset?
For logging, I have a separate video in the pipeline, You would rather export logs from the app and not store it in a volume. Take a look at the EFK\ELK concepts, where basically every pod in K8s writes logs to stdout. It is best practise not to write logs to file and rather use stdout\stderr. These logs will automatically end up in the host docker logs which you can push out of the system using Fluentd\FluentBit. This allows you to centralise logs for all deployments\statefulsets running in your cluster under something like Elastic Search or 3rd party application.
@@MarcelDempers oh. interesting. I'll look it up! Thank youuu so much. been watching your videos. so much helpful for me. great videos btw :)))
Hi Thanks you for great video.! I did the same and tried to deploy jenkins but when i create pv and pvc . mnt/data is empty. and i used volumemount to var/jenkins_home.
If you need help with Jenkins checkout my CI/CD playlist, there are two Jenkins guides with source code for you 💪🏽👊🏽
but If we have a three node cluster one is master other two are worker first time our Postgres pod is schedule on worker1 now its PV on worker1 and second time after deleting the pod k8s schedule it on Worker2 now pod create its PV on Worker2 but now data is not persistence. because data on worker1 can not allocate to worker2 pod? What's a solution?
This is because you are using a deployment with PVs which is not meant for stateful workloads. You'll need to learn about StatefulSets
ruclips.net/video/zj6r_EEhv6s/видео.html
Hey Marcel, great videos. After I altered the ConfigMap to use a different POSTGRES_USER and deleted/recreated everything including the PV, the old database user & table existed.
Can you explain why that happened even after I deleted the PV? Is there more to "scrub" here?
I assumed recreating everything would result in a fresh database with the new user credentials. Thanks!
This highly depends on the storage class you use. In this demo I use host path which means data is persisted on the node at a path (which can be changed).
The get a fresh storage, you either have to change the path to get a new folder on the host. Remember your old data would remain. Or SSH to the node, clean it up, Or run another container with a host mount and carefully run "rm" command to delete the content
@@MarcelDempers Thank you for such a quick response! I used hostPath/your tutorial code too. If you have any advice on ssh-ing to the docker-desktop (Mac) node to clean up, I'm all ears. Thanks again for these videos, very helpful.
Hi Marcel, my setup is a bit unique, I have created a K8S cluster on a few VMs in our own private cloud and all the nodes are mounted with a single NFS drive (say /data), can I define a storageclass of hostpath type using /data? Does PV has to be mounted to all the nodes?
That depends on the app you're planning to run and how it uses the storage. If you plan on using a statefulset its recommended that each pod has its own storage. StatefulSet will use a storage class to provision a PV per pod and not share the same storage. Hostpath will work, but if you mount a shared storage to all the nodes, they may override each other's data. So unless you mount unique storage to each node, it may not be ideal. I'm new to NFS, but i would see if its possible to provision NFS dynamically using a storage class and see if the statefulset can issue PV + PVCs per pods using NFS dynamic provisioning. That way each pod will get its own NFS storage provisioned by k8s when you scale up and down
Checkout NFS provisioning using Rook rook.io/docs/rook/v1.4/nfs.html
Seems interesting.
If you have not worked with Statefulsets I'd highly recommend to ruclips.net/video/zj6r_EEhv6s/видео.html
@@MarcelDempers thanks Marcel, I have deployed Rook Ceph RBD in my cluster and now the applications now works like charm
Hey Marcel, once again thanks for another excellent video. One question though. Would the Postgres data survive a Docker Desktop restart?
Technically, it should since the example used host path persistence so unless the host is destroyed, it should persist yes
@@MarcelDempers Hey Macel, thanks for the video. unfortunately the data being lost when docker restarts.
@@MarcelDempers data loss after restarting the docker desktop
Why does the table already exist @4:45 if you just created the volume. I'm guessing you edited out the part when you first created it.
You requested for 50mb you got 1gb :P :D
I believe capacity stated on the PVC is that of the PV and not the requested of the claim :)
Do people really prefer the background music ? Maybe you should poll your subscribers, just distracting.
Great video!