I think I said it in one of you first videos but again, you are the best DevOps educator on youtube. Clearly you've got experience that many others don't. Your vids aren't some robotic read-out of the official docs for views, so I can always find something new. Great style, great structure. You definitely deserve more recognition but it's getting there and certainly it will get there!
I will rate you 100% . The content was too notch and the explanation was just what I need now. I wanted to deploy a postgres database on k8s, so I sort for resources on data persistent and I couldn't find any that is more explanatory. Your sample that resides in github was top-notch.
Your vedios are really helpful and i love the the way you explain it with real time use cases. Would request you to cover helm charts from scratch to complex. Lets say how to create helm charts, how to convert our own applications to helm charts etc etc. I would love to have some series of vedios on this with the ongoing standard use cases which currently organisations are doing. Thanks! keep the great work going going. Happy learning!!
Thanks to you, I learned a lot of "production / professional" ways to understand and implement those strategies. None of your videos get bored, I even watch some vid 5-6 times to strengthen my knowledge until I completely understand everything. Thank you for your supports.
I don't know what I would do without your channel. You have an amazing skill to explain complicated topics in such a wonderful, easy-to-understand way. Keep going!! A video with Tekton would be great, if I can propose something.
StatefulSets are an interesting beast. When I first ran into them, I didn't know what to make of it : I've always thought of k8s as stateless-oriented and StatefulSets felt like an anti-pattern. Once I looked into clustered / high availability databases and the on-going work in that space, it made a lot more sense (I'm a big fan of Stolon, if you're into distributed Postgres). K8s is really a great engine to distribute *stuff*, it doesn't have to be microservices.
Outstanding video mate. I'd say you are a welcome addition to our sunny shores but you chose VIC instead of WA :P Just teasing Melbourne rocks, lived there for 4 years myself as a young lad.
The way you explain concepts and then finish them with real world examples is what most engineers wish to see. Redis, RabbitMQ I have watched both and request you to deploy more such complex applications. Also, if you can tell us what could be the best setup for cluster monitoring and how to approach to solve real world cluster down scenarios, that would be great.
Your video on the topic is excellent!! I would love to see more examples of statefulsets because your video has raised some critical issues about cluster failure which couldnt be automatically resolved by the kubernetes engine and may require manually fixing the cluster setup. Can you please show possible solutions on fixing cluster issues that cannot be automatically resolved by the k8s engine?
just rewatched this... thinking I'm picking up on the complexity of stateful sets and ... a stretched environment like multiple AZ's as present on AWS EKS... any chance you've done a video, where say 3 or more stateful sets are pined to nodes (AWS EKS Nodegroups). this can then be used against the Redis deployment and PostgreSQL deployment ?
My Redis series does this already. A statefulset will pin a pod to the same node's volume claim. Unless that node disappeared, the pod will always stay on that node. If a rolling node upgrade occurs which my roll new VMs, you may lose data ( depending how the cloud provider does updates)
@@MarcelDempers understand the stateful pod will gravitate back to where the pvc is, but i want to pin/direct the pod before that, I have a EKS cluster spread across 3 x az's, and additionally the cluster have node groups for apps and nodegroups for db... i want to direct the stateful set to the database node groups, and additionally push each of the Redis masters onto a node in each of the az's.
I would like to add a correction, please correct me if I'm wrong. At 12:18 you said in deployment scaling the pods scale all at same time but I think with rolling updates we can choose % of scaling pods at a time.
This is an awesome series, but I wanted to point out an issue with the command @22:28. The command adds an extra ":6379" at the end of the string which is recorded in the env var. The output from: kubectl -n example get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ' in my case was: "10.244.0.6:6379 10.244.0.8:6379 10.244.0.10:6379 10.244.0.12:6379 10.244.0.14:6379 10.244.0.16:6379 :6379" This causes the next cluster create command to fail. Took me a little bit to understand why the command was failing if anyone else runs into the issue.
In your example you used ClusterIP type of service. Is headless service not one of requirements for statefulset? Or I can still use ClusterIP? Good video anyway, thanks.
another good videos on Kubernetes.. can you please make video to capture logs of application in cabernets in ELK with cabana or any other tools to view app logs
Hello Marcel, Can you make a video based on DevOps security aspects ?.. Like all things to be considered when we think about security in CICD? Like as a hiring manager what all u think is important and relevant.
Hello, thank you for the tutorial ! I've got some questions at 25:30 you show that PV are retained but the reclaim policy is set to Delete. How does it work ? Does it only occur when the StatefulSet get fully deleted ?
Great question. As per documentation, For reclaim policy "Delete", the volumes are only deleted when the PVC's are deleted. Current statefulset behaviour is PVC's are not deleted with pods or the statefulsets are deleted since they have a different lifespan.
Learning about PVs and noticed that all of the PVs deployed into our cluster have the reclaim policy = Delete, which had me concerned that our engineering team had possibly made a terrible oversight in the deployments. This comment is exactly what I wanted to understand. Thank you so much for this comment and the excellent statefulset tutorial!
Hey, do you have a video describing how to make an EBS based StatefulSet in AWS? It seems that there's an issue reguarding EBS volumes being bound to an AZ. I heard that I should create an ASG per AZ, and make my StatefulSet have Pods across the AZs, but I don't know how to enforce that behavior. Most of my volumes are EFS and FSx so far, but I'd like to run a database in EKS, which would need EBS for speed, and I don't want to miss anything.
Hi Thanks for the brilliant content! Does the redis deployment automatically distribute across six note? If I have a taint on master nodes I guess that will be an issue? Cheers
Thank you so much for your video. I have a question about scaling down Cassandra cluster, as you know, Cassandra cluster will store all the data in different databases, let's assume we have 5 Cassandra databases with replica of 3, and then scaling down to 1 Cassandra database, how can we know if the left Cassandra database consists all the data of the 5 Cassandra databases. Or it is not possible to do scaling down Cassandra like Redis in k8s? Thank you very much
Hi At first would like to thank you for the wonderful videos...I am having request... Could you please make a detailed video on how MySQL High availability works and how to configure it on k8s. Also detailed video from. Basic to Advance on K8S Networking.. Thanks a lot in Advance..Stay Safe..!!
Hi, the yamls shown in the video are different from the git. For instance the service type for redis cluster used in the video is ClusterIP but in git its headless service. Also why service name is to be mentioned in statefulset definition?
I was looking over the Github repo for the redis stateful set. I have a question. If each replica is mounting to the same host path is there any chance for mounting collisions and data corruption?
definitely helped explain stateful sets and pv's snd pvc's... but now to map to real world... 2 AZ's on AWS with Prometheus deployed, with a ALB in front ? if the Prometheus servers were running on the same node I'd say lets use deployments, that way either node would have access to all the data, but then you going to have duplication where they scrape the same target. lets make it simpler. one prometheus server, running active in AZ1.. but it needs to be protected to be able to fail over to AZ2... snd as I discovered EFS not allowed.
hello Marcel !! could you please let me know how to troubleshoot based on the error at the end of ur video ( scale down to zero - and then making up the replicas , but still the cluster is down ) how to overcome this ?
It's an interesting issue right ? :) This is part of the reason I recommend SaaS solutions for storage. I myself am not a Redis expert, so I would need to dig into 1) Redis cluster best practises, 2) Redis cluster troubleshooting \ repair process. I am planning a Redis series in the future 💪🏽
Good question. I have not tried it. The statefulset spec.serviceName might have some mandatory requirements, but nothing stops you from creating another clusterIP service or LoadBalancer one and selecting the statefulset pods with a label selector, if you need to
Hello, I use microk8s and run $IPs = $(microk8s.kubectl -n example get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ') I got =: command not found How can I solve this problem?
Hi Marcel, Commenting on old video. But recently there has been a surge in peopel running their DBs as STS. What you make of it? Personally - I still find running DB in HA/cluster is pretty much hectic. Unless you're master of DB & k8s...makes no sense to containerize your DB. I even treid your postgres series. Tried even part 5 (for which you haven't made any video...but github docs are there) you have used two stateful sets with replication between them. And each STS had only 1 replica. You didn't bother using any operators (or helm) for deplying HA postgres on k8s. And that video is hardly year old. So I assume you stil are of the opinion about not running DB are STS.
I dont think there is a clear right or wrong answer with DBs on STS on k8s. I think you understand the pitfalls to make an informed decision. For me, the series is all about learning the complexities of database sync and its internal workings so it helps people be more informed when running postgres and other dbs on k8s. Also i planned to try out the helm operators so i could understand them better after building it out manually. I personally run a couple of types of dbs like redis caches and message queues on k8s but this is mainly because they are somewhat stateless, if they die, the platform is not down and there would be limited customer impact. For true state, i personally prefer paying for a cloud based DB as the cost is way more justified. The cloud provider takes care of the complexities and your team can focus on k8s (which already has complexity) You dont want to stretch engineering teams too thin so theres always places where id recommend Do it yourself and others where its like "just pay the couple hundred extra and get peace of mind" Hope that helps
@@MarcelDempers Even I used rabbitmq operator and created STS for rabbitmq cluster. Loosing message broker is not show killer and doesn't bring down application. But we are still using machines for postgres (with streaming replication with pgbackrest)
You are talking too fast sometimes, when I watch yours videos i need to pause coule of times and rewind 20 sek to catch everything I need. But sometimes there is a lack of details for example how this web app is connecting to redis in this example
Dude, your series on k8s is - by far - the celarest, most concise and illuminating set of vids out there. I sincerely commend you, sir.
I am on the virtuous path of watching every single video of this channel. An practice it. So far it is going great and I have learned a ton.
Greatly explained . Currently working as DevOps intern and really was stuck with this statefulset and deployment. keep up good work
I think I said it in one of you first videos but again, you are the best DevOps educator on youtube. Clearly you've got experience that many others don't. Your vids aren't some robotic read-out of the official docs for views, so I can always find something new. Great style, great structure. You definitely deserve more recognition but it's getting there and certainly it will get there!
Very helpful. Thanks for the videos with a good emphasis on the "Why" part.
Thank you for your awesome videos, one of the best series of videos on K8s on youtube. TYSM.
you, sir, are answering all the exact questions which no one other answers. thank you.
I will rate you 100% . The content was too notch and the explanation was just what I need now.
I wanted to deploy a postgres database on k8s, so I sort for resources on data persistent and I couldn't find any that is more explanatory.
Your sample that resides in github was top-notch.
Now that's called pure gold 💰 . Awesome stuff mate ! 👏
Your vedios are really helpful and i love the the way you explain it with real time use cases. Would request you to cover helm charts from scratch to complex. Lets say how to create helm charts, how to convert our own applications to helm charts etc etc. I would love to have some series of vedios on this with the ongoing standard use cases which currently organisations are doing. Thanks! keep the great work going going. Happy learning!!
Man, your videos are awesome. Learnt a lot from it!
Thanks to you, I learned a lot of "production / professional" ways to understand and implement those strategies. None of your videos get bored, I even watch some vid 5-6 times to strengthen my knowledge until I completely understand everything. Thank you for your supports.
I don't know what I would do without your channel. You have an amazing skill to explain complicated topics in such a wonderful, easy-to-understand way. Keep going!!
A video with Tekton would be great, if I can propose something.
You are a great instructor and cross Nana 👏
A very clear explanation, thanks!
It’s very clear and concise.. really appreciate it..
You can explain very good and in this way many others can understand this technologies
I loved this stateful sets explanation
very good in-depth explanation. just what was looking for. thanks
stateful explained in a very clear way. Thank you for the video.
You are best keep sharing the concept and. Keep it simple as other devops video are not easy to understand on another channels
MVP Channel! 10x Marcel 🙏
Love ur content! Keep on rockin'
Very Very helful vedio and was very easy to understand StatefulSet. Thanks you so much...
Peace Marcel, You are the man!
you are awsom! thank you the DevOps guy))
Best of the best, you are the man, thanks a lot !
This is a very very good tutorial. Thank you and subscribed!
You're the best man, this video is amazing as always
StatefulSets are an interesting beast. When I first ran into them, I didn't know what to make of it : I've always thought of k8s as stateless-oriented and StatefulSets felt like an anti-pattern. Once I looked into clustered / high availability databases and the on-going work in that space, it made a lot more sense (I'm a big fan of Stolon, if you're into distributed Postgres). K8s is really a great engine to distribute *stuff*, it doesn't have to be microservices.
super well explained! thank you!
Thanks so much for the content!
it alway nice watch your videos ! best ever!
Outstanding video mate. I'd say you are a welcome addition to our sunny shores but you chose VIC instead of WA :P
Just teasing Melbourne rocks, lived there for 4 years myself as a young lad.
The way you explain concepts and then finish them with real world examples is what most engineers wish to see. Redis, RabbitMQ I have watched both and request you to deploy more such complex applications. Also, if you can tell us what could be the best setup for cluster monitoring and how to approach to solve real world cluster down scenarios, that would be great.
Your video on the topic is excellent!! I would love to see more examples of statefulsets because your video has raised some critical issues about cluster failure which couldnt be automatically resolved by the kubernetes engine and may require manually fixing the cluster setup. Can you please show possible solutions on fixing cluster issues that cannot be automatically resolved by the k8s engine?
Nice explanation! Keep up the good work!
This is helpful .. thanks!
Awesome explanation! Thank you.
That was master class :)
Nice work!
holy crap! what a video... awesome!
Ur arms are ripped dude
Very well laid out. Great job!
This fellow is no mere mortal
Awesome. Thank you!
just rewatched this...
thinking I'm picking up on the complexity of stateful sets and ... a stretched environment like multiple AZ's as present on AWS EKS... any chance you've done a video, where say 3 or more stateful sets are pined to nodes (AWS EKS Nodegroups).
this can then be used against the Redis deployment and PostgreSQL deployment ?
My Redis series does this already. A statefulset will pin a pod to the same node's volume claim. Unless that node disappeared, the pod will always stay on that node. If a rolling node upgrade occurs which my roll new VMs, you may lose data ( depending how the cloud provider does updates)
@@MarcelDempers understand the stateful pod will gravitate back to where the pvc is, but i want to pin/direct the pod before that, I have a EKS cluster spread across 3 x az's, and additionally the cluster have node groups for apps and nodegroups for db... i want to direct the stateful set to the database node groups, and additionally push each of the Redis masters onto a node in each of the az's.
Informative video . Keep on Going with these type of videos.
Thanks a lot for your videos! It is extremely helpful!
I would like to add a correction, please correct me if I'm wrong. At 12:18 you said in deployment scaling the pods scale all at same time but I think with rolling updates we can choose % of scaling pods at a time.
Could you please make a video on stateless set
Best video ever...
Your Just Awsome!!! Very helpful.
This is an awesome series, but I wanted to point out an issue with the command @22:28. The command adds an extra ":6379" at the end of the string which is recorded in the env var.
The output from:
kubectl -n example get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 '
in my case was:
"10.244.0.6:6379 10.244.0.8:6379 10.244.0.10:6379 10.244.0.12:6379 10.244.0.14:6379 10.244.0.16:6379 :6379"
This causes the next cluster create command to fail. Took me a little bit to understand why the command was failing if anyone else runs into the issue.
Great explanation.
In your example you used ClusterIP type of service. Is headless service not one of requirements for statefulset? Or I can still use ClusterIP? Good video anyway, thanks.
THE BEST!
another good videos on Kubernetes.. can you please make video to capture logs of application in cabernets in ELK with cabana or any other tools to view app logs
Thanks for the kind words 💪🏽
We have started that journey already 🤓
Come follow along ruclips.net/p/PLHq1uqvAteVvfDxFW50Mdezk0xum-tyHT
Does statefulsets already includes data & file replication?
Hello Marcel,
Can you make a video based on DevOps security aspects ?.. Like all things to be considered when we think about security in CICD? Like as a hiring manager what all u think is important and relevant.
Good job mate! it easy to digest ;)
love ur content please make a video on Daemonset
Great content. Thank you!
Hello, thank you for the tutorial ! I've got some questions at 25:30 you show that PV are retained but the reclaim policy is set to Delete. How does it work ? Does it only occur when the StatefulSet get fully deleted ?
Great question. As per documentation, For reclaim policy "Delete", the volumes are only deleted when the PVC's are deleted. Current statefulset behaviour is PVC's are not deleted with pods or the statefulsets are deleted since they have a different lifespan.
@@MarcelDempers thank you
Learning about PVs and noticed that all of the PVs deployed into our cluster have the reclaim policy = Delete, which had me concerned that our engineering team had possibly made a terrible oversight in the deployments.
This comment is exactly what I wanted to understand. Thank you so much for this comment and the excellent statefulset tutorial!
Hey, do you have a video describing how to make an EBS based StatefulSet in AWS? It seems that there's an issue reguarding EBS volumes being bound to an AZ. I heard that I should create an ASG per AZ, and make my StatefulSet have Pods across the AZs, but I don't know how to enforce that behavior. Most of my volumes are EFS and FSx so far, but I'd like to run a database in EKS, which would need EBS for speed, and I don't want to miss anything.
Great explanation!
Hi Thanks for the brilliant content! Does the redis deployment automatically distribute across six note? If I have a taint on master nodes I guess that will be an issue? Cheers
Wonderful - Well explained - does this apply to Kubernetes version 1.20 with containerd ?
Please let me know which software you are using for diagram
Thanks for your support on kubernetes..
My request could you please make zuul api gateway for kubernetes cluster for routing
Thank you so much for your video. I have a question about scaling down Cassandra cluster, as you know, Cassandra cluster will store all the data in different databases, let's assume we have 5 Cassandra databases with replica of 3, and then scaling down to 1 Cassandra database, how can we know if the left Cassandra database consists all the data of the 5 Cassandra databases. Or it is not possible to do scaling down Cassandra like Redis in k8s? Thank you very much
Hi At first would like to thank you for the wonderful videos...I am having request... Could you please make a detailed video on how MySQL High availability works and how to configure it on k8s. Also detailed video from. Basic to Advance on K8S Networking.. Thanks a lot in Advance..Stay Safe..!!
how do we make cluster of multiple nodes using docker for windows?
super thanks!!!
Hi, the yamls shown in the video are different from the git. For instance the service type for redis cluster used in the video is ClusterIP but in git its headless service. Also why service name is to be mentioned in statefulset definition?
Very helpful :)
I was looking over the Github repo for the redis stateful set. I have a question. If each replica is mounting to the same host path is there any chance for mounting collisions and data corruption?
At least the subPath on the host needs to be unique. Each pod should have it's own data, yes
Checkout the Redis playlist on the channel for a more in-depth Redis setup
definitely helped explain stateful sets and pv's snd pvc's... but now to map to real world... 2 AZ's on AWS with Prometheus deployed, with a ALB in front ?
if the Prometheus servers were running on the same node I'd say lets use deployments, that way either node would have access to all the data, but then you going to have duplication where they scrape the same target.
lets make it simpler. one prometheus server, running active in AZ1.. but it needs to be protected to be able to fail over to AZ2...
snd as I discovered EFS not allowed.
How do we fix that cluster outage then? 🤔
Is it ok to use pod ip's for clustering? I think If pods get rescheduled their pod ip addresses will change and break the redis cluster.
Correct, you dont want to use Pod IPs. You'll want to use K8s headless services with statefulsets. Take a look at my Redis playlist as an example.
Peace! ☮
How does the redis cluster keep track of the new pod members when you scale up the statefulset ?
Checkout my Redis guide which includes a video on HA and Kubernetes
ruclips.net/p/PLHq1uqvAteVtlgFkmOlIqWro3XP26y_oW
hello Marcel !! could you please let me know how to troubleshoot based on the error at the end of ur video ( scale down to zero - and then making up the replicas , but still the cluster is down ) how to overcome this ?
It's an interesting issue right ? :) This is part of the reason I recommend SaaS solutions for storage. I myself am not a Redis expert, so I would need to dig into 1) Redis cluster best practises, 2) Redis cluster troubleshooting \ repair process. I am planning a Redis series in the future 💪🏽
@@MarcelDempers Thank you for great video. Have you come up with any solution for this issue? Is Redis Sentinel answer for this issue?
If I get the job I will send you a beer !
Is it mandatory to use headless service with StatefulSet?
Good question. I have not tried it. The statefulset spec.serviceName might have some mandatory requirements, but nothing stops you from creating another clusterIP service or LoadBalancer one and selecting the statefulset pods with a label selector, if you need to
You are beautifully pumped man. But please don't jump so quick from example code to yourself.
Nice videos. Thank you!!!
Perfect
Hello,
I use microk8s and run
$IPs = $(microk8s.kubectl -n example get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')
I got
=: command not found
How can I solve this problem?
I'm having the same problem
first. can you please do a scheduler video?
Sweet
👏
best
Hi Marcel,
Commenting on old video. But recently there has been a surge in peopel running their DBs as STS. What you make of it?
Personally - I still find running DB in HA/cluster is pretty much hectic. Unless you're master of DB & k8s...makes no sense to containerize your DB. I even treid your postgres series. Tried even part 5 (for which you haven't made any video...but github docs are there) you have used two stateful sets with replication between them. And each STS had only 1 replica. You didn't bother using any operators (or helm) for deplying HA postgres on k8s. And that video is hardly year old. So I assume you stil are of the opinion about not running DB are STS.
I dont think there is a clear right or wrong answer with DBs on STS on k8s. I think you understand the pitfalls to make an informed decision. For me, the series is all about learning the complexities of database sync and its internal workings so it helps people be more informed when running postgres and other dbs on k8s. Also i planned to try out the helm operators so i could understand them better after building it out manually.
I personally run a couple of types of dbs like redis caches and message queues on k8s but this is mainly because they are somewhat stateless, if they die, the platform is not down and there would be limited customer impact. For true state, i personally prefer paying for a cloud based DB as the cost is way more justified. The cloud provider takes care of the complexities and your team can focus on k8s (which already has complexity) You dont want to stretch engineering teams too thin so theres always places where id recommend Do it yourself and others where its like "just pay the couple hundred extra and get peace of mind"
Hope that helps
@@MarcelDempers Even I used rabbitmq operator and created STS for rabbitmq cluster. Loosing message broker is not show killer and doesn't bring down application. But we are still using machines for postgres (with streaming replication with pgbackrest)
great channel - it would be better just to show the screen though - the montage is too hectic
Dude, I really like your efforts, I've learned alot. No offense but whenever I see you I think It's a murderer trying to teach k8s :D
You are talking too fast sometimes, when I watch yours videos i need to pause coule of times and rewind 20 sek to catch everything I need. But sometimes there is a lack of details for example how this web app is connecting to redis in this example
you are beautiful
Amazing video! thank you!
Great video, thanks!
Excellent video! Thank you!