When a leader for a partition goes down and a new leader is elected. 1) In between that time what happens to read and writes for that partition . 2) what happens to messages above high watermark
If the leader is down client reads and writes fail. The client has no connection and must reconnect to an available broker. If the broker is up but is not the leader the client gets "not the leader" exceptions. The recovery action is to find the new leader and connect to it. The more current apis hide these details.
Mag has answered the 1st question, I will answer the 2nd. The messages above the high watermark will get lost. This may seem data-loss but the Producer is responsible to tune the durability of their messages during the write with more replica acks, so with leader + 1 replica ack the producer will have durability with at-most 1 node failure, with leader + 2 replica ack the producer will accommodate 2 node failures, with just leader ack there can be data-loss even when 1 node i.e the leader dies
If Ted Lasso is giving a tech talk, you have to watch it for sure! Amazing! Thanks for sharing this! 10/10 Awesome Talk.
This is how I imagine, Ted Lasso would give a tech talk. 10/10 Excellent Talk.
Tim as always a great speaker.
Loved the talk!
Summarised all the concepts very well!
Loved ISR high water mark lol 🤣
Great speaker and great talk.. I would love to see him give more talk about this topic.
Very interesting talk for beginners ! Thanks !
Exceptional talker, good stuff
Best Kafka talk!
good enough
Excellent talk!
excellent talk
When a leader for a partition goes down and a new leader is elected.
1) In between that time what happens to read and writes for that partition .
2) what happens to messages above high watermark
If the leader is down client reads and writes fail. The client has no connection and must reconnect to an available broker. If the broker is up but is not the leader the client gets "not the leader" exceptions. The recovery action is to find the new leader and connect to it. The more current apis hide these details.
Mag has answered the 1st question, I will answer the 2nd.
The messages above the high watermark will get lost. This may seem data-loss but the Producer is responsible to tune the durability of their messages during the write with more replica acks, so with leader + 1 replica ack the producer will have durability with at-most 1 node failure, with leader + 2 replica ack the producer will accommodate 2 node failures, with just leader ack there can be data-loss even when 1 node i.e the leader dies
41.25
Not impressed by this dude's speaking style. Less dry humor, more content please.