nice explanation,, lets say i have 5 worker node and my writer pod scheduled at worker 3 with persistence volume dynamically ,, if that worker node goes down what will happen ..
We Create a Headless service ( by setting clusterIP: None). This assigns all the pods a unique DNS name instead of just a cluster-level IP. The mysql-0 dns name is always the writer because we write this logic in the statefultsets configuration file (if index == 0, apply master config else apply reader config) kubernetes.io/docs/concepts/services-networking/service/#headless-services
@sre101: can you suggest storage class which used and also suggest if you increase replica then seperate Pv will be provisioned as per value defined in pvc, and also suggest how we can do in AWS if we want to keep seperate Pv for each project then how
Storage class really depends on the nature of your data and what matters more to you. eg- If you're storing payment transactions data, you want very strong guarantees that this data will not be lost, even if the latency is a bit high. You want backups and disaster recovery on it. You probably also want the database to be CP (in terms of CAP theorem). So your storage class would reflect those needs.
@@sre101: can you please create a video or suggest as per my queries: 1: which storage class you used?, 2:if we want 3 replicas, then we would need to create pvc definition for each members in yaml files.? 3: how we can do in AWS if we want to keep seperate PV for each project?
@@Kk-rl7nv 1. Because I'm using minikube, I'm using the csi-hostpath-driver (see minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/) 2. No, that's not scalable. Instead, you can use VolumeClaimTemplate (github.com/duaraghav8/SRE101/blob/main/run-db-in-k8s/statefulsets.yaml#L157) 3. I believe you would need an appropriate Volume plugin to use AWS storage. Maybe this article can help you - aws.amazon.com/blogs/storage/persistent-storage-for-kubernetes/
do we store the data of database pods in another instance outside of the cluster like aws rds or we just store everything in those storage classes itself in production?
Depends on the nature of your data. Normally, people want high durability guarantees for their production data. So storing them on local machine disks is not a great strategy. So for example, your Persistent Volume could connect to AWS EBS or S# under the hood and whenever your app writes some data to this PV, it is actually being written to the AWS storage. If you want to store structured data in AWS RDS, you don't need a PV. In that case you application directly connects to the RDS database using its endpoint and credentials.
for HA - deploy more read replica pods across different worker nodes (or even different datacenters, regions) for HA for writer, you'll need to explore sharding. I haven't implemented auto failover myself yet, so can't comment much about it.
@@hamzaerrahma9858 I ran it in production for a few weeks but for personal use. There were no restarts unless mysql itself crashed (mostly due to resource constraints for me)
Hey Thanks for the feedback! I agree, I'm planning a part 2 for this where I cover more advanced topics like autoscaling, backups & disaster recovery, performance tuning, etc.
Great question. The writer Pod's scaling mainly depends on how you scale the writer node of MySQL. IMO you have 2 options: 1. Vertical scaling: Keep adding more CPU, Memory, Disk, Network bandwidth resources to your writer machine until you reach the most expensive machine and any further addition only degrades the performance. With vertical, you still don't have high availability in writer. 2. Horizontal scaling: Add more writer nodes and distribute the write queries among them. This can be accomplished using Sharding. Some tricks you can also use: 1. people use to scale writer is to direct all read queries to read replicas so that the writer only needs to deal with write requests, so it can give higher throughput. 2. Use faster storage disk to increase data write speed. this can increase throughput.
In the video, the headless Service "mysql" is bound to the StatefulSet "mysql". This StatefulSet is also bound to the Service "mysql-read"? I mean the "serviceName" in the StatefulSet is set to "mysql" which is the headless service, not "mysql-read".
Thanks for your concise and informative illustration. It is so enlightening for me as a Kubernetes beginner!
Thanks a lot for your kind words :)
loved this video!! subbed right away!!
thank you!
Excellent Video...Please keep doing the good work. 👍👍
Glad you found it useful :)
great content and video 👍
Thank you Pankaj, means a lot
nice explanation,, lets say i have 5 worker node and my writer pod scheduled at worker 3 with persistence volume dynamically ,, if that worker node goes down what will happen ..
Hi brother, did you implement that? Are the pods running successfully without restarts?
Hi, I didn't get how replica set gets only pod's IP instead of service IP. Could you please explain in detail?
We Create a Headless service ( by setting clusterIP: None). This assigns all the pods a unique DNS name instead of just a cluster-level IP.
The mysql-0 dns name is always the writer because we write this logic in the statefultsets configuration file (if index == 0, apply master config else apply reader config)
kubernetes.io/docs/concepts/services-networking/service/#headless-services
could you provide me the code for statefull set
github.com/duaraghav8/SRE101/blob/main/run-db-in-k8s/statefulsets.yaml
@sre101: can you suggest storage class which used and also suggest if you increase replica then seperate Pv will be provisioned as per value defined in pvc, and also suggest how we can do in AWS if we want to keep seperate Pv for each project then how
Storage class really depends on the nature of your data and what matters more to you.
eg- If you're storing payment transactions data, you want very strong guarantees that this data will not be lost, even if the latency is a bit high.
You want backups and disaster recovery on it.
You probably also want the database to be CP (in terms of CAP theorem).
So your storage class would reflect those needs.
@@sre101: can you please create a video or suggest as per my queries: 1: which storage class you used?, 2:if we want 3 replicas, then we would need to create pvc definition for each members in yaml files.?
3: how we can do in AWS if we want to keep seperate PV for each project?
@@Kk-rl7nv
1. Because I'm using minikube, I'm using the csi-hostpath-driver (see minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/)
2. No, that's not scalable. Instead, you can use VolumeClaimTemplate (github.com/duaraghav8/SRE101/blob/main/run-db-in-k8s/statefulsets.yaml#L157)
3. I believe you would need an appropriate Volume plugin to use AWS storage. Maybe this article can help you - aws.amazon.com/blogs/storage/persistent-storage-for-kubernetes/
do we store the data of database pods in another instance outside of the cluster like aws rds or we just store everything in those storage classes itself in production?
Depends on the nature of your data.
Normally, people want high durability guarantees for their production data. So storing them on local machine disks is not a great strategy.
So for example, your Persistent Volume could connect to AWS EBS or S# under the hood and whenever your app writes some data to this PV, it is actually being written to the AWS storage.
If you want to store structured data in AWS RDS, you don't need a PV. In that case you application directly connects to the RDS database using its endpoint and credentials.
i need to have high availability and auto failover for master.
What needs to be done here?
Hi brother, did you implement that? Are the pods running successfully without restarts?
for HA - deploy more read replica pods across different worker nodes (or even different datacenters, regions)
for HA for writer, you'll need to explore sharding.
I haven't implemented auto failover myself yet, so can't comment much about it.
Good learnings, thanks for the wonderful video 😊
🙌
Hi brother, did you implement that? Are the pods running successfully without restarts?
@@hamzaerrahma9858 I ran it in production for a few weeks but for personal use. There were no restarts unless mysql itself crashed (mostly due to resource constraints for me)
Thanks @Raghav for a clear explanation. Would like see more videos on DB scaling dynamically in K8s.
Hey Thanks for the feedback! I agree, I'm planning a part 2 for this where I cover more advanced topics like autoscaling, backups & disaster recovery, performance tuning, etc.
How can i do this with an NFS server instead of stroring data locally
Thank you for the very useful video! Can you explain how the writer pod can be scaled if at all?
Great question. The writer Pod's scaling mainly depends on how you scale the writer node of MySQL.
IMO you have 2 options:
1. Vertical scaling: Keep adding more CPU, Memory, Disk, Network bandwidth resources to your writer machine until you reach the most expensive machine and any further addition only degrades the performance.
With vertical, you still don't have high availability in writer.
2. Horizontal scaling: Add more writer nodes and distribute the write queries among them. This can be accomplished using Sharding.
Some tricks you can also use:
1. people use to scale writer is to direct all read queries to read replicas so that the writer only needs to deal with write requests, so it can give higher throughput.
2. Use faster storage disk to increase data write speed. this can increase throughput.
In the video, the headless Service "mysql" is bound to the StatefulSet "mysql". This StatefulSet is also bound to the Service "mysql-read"? I mean the "serviceName" in the StatefulSet is set to "mysql" which is the headless service, not "mysql-read".
The StatefulSet is bount to the "mysql" service because that;s the main service controlling the cluster.