Thank you for watching. Also, NEW EPISODE: interviewready.io/learn/system-design-course/building-an-ecommerce-app-1-to-1m/7-the-case-of-the-expensive-cache
Have been following you since long and i am so impressed with the consistency and humility. Yoir explanations are simple and you seem like a college mate we are getting to learn from.thanks for all the free content
It’s a great video. It covered the features of kafka well. I feel the way you explain a problem and explain how to overcome it might not be obvious to the listeners that kafka automatically takes care of these. For eg, you said multiple consumers might consume a message from the same influencer topic leading to notification duplication, and you explained consumer grouping as the solution to it. But you might have missed to explain that it is not actually possible, any extra consumers more than one active on a partition sits idle and kafka takes care of it automatically. Viewers might think that as some inherent problem with kafka and they would be required to handle it. Apart from these minor things the video is very good. Thanks for making it!
6:58 If I remember correctly, ZooKeeper uses the Zab (ZooKeeper Atomic Broadcast) protocol instead of Paxos for consensus. Starting with version 2.8, Kafka no longer relies on ZooKeeper for leader election. Instead, it uses KRaft (Kafka Raft), Kafka’s implementation of the Raft consensus algorithm, for managing metadata and coordination internally.
Hi @gkcs, it looks Kafka partition leader handles both read & write. Write Acknowledgments: The level of acknowledgment (acks) from the leader to the producer depends on the producer configuration: acks=0: The producer does not wait for acknowledgment. acks=1: The producer waits for acknowledgment only from the leader. acks=all: The producer waits for acknowledgment from all in-sync replicas (ISR). [ Read ] Consistency: For strict consistency, consumers can be configured to only read from committed offsets (data replicated to all in-sync replicas). Also newer version of Kafka (2.8 onwards) introduced KRaft (Kafka Raft) as replacement for Zookeeper to simplify operations & system design.
1) Why do you call yourself gkcs 2) These new videos you upload, do you add them to your system design course as well? (I have enrolled and I am trying to figure out if your course is enough, or if I need to watch the videos as well) Great video as always btw 😁
1. It stands for General Knowledge Counter Strike. It was my cs handle in college, and it stuck with the RUclips channel :D 2. We typically go into more depth at InterviewReady with the research papers. We dive into the papers as a group during the live zoom sessions. You can see the recordings here: interviewready.io/learn/system-design-course/scaling-memcached-at-facebook/november---memcached-reading Thank you!
I asked the first same question years ago, I initially thought it was General Knowledge Computer Science. @gkcs put up a poll after that If I remember correctly, and then I got to know the real full form
At 5:36, why didn't the message no. 4 get replicated to the other partitions after it was received?? That way, when middle kafka broker would receive 5, everything would remain consistent
that was just a problem scenario for explaining possible data inconsistency which doesn't happen ... if you set acks to leader or min sync replicas to 1 this can still happen
Well explained, I’m a Frontend developer and heard people talking about Kafka so this helps me to be part of discussion. Also, can you complete the Application cache box at 13:18. It triggered my OCD 😂
I see you have mentioned multiple time that kafka consumer reads from replicas. But I guess this is incorrect, both producer and consumer only interact with leader at any point of time. Can you confirm this? You are right where you mentioned "Consumer consume message only once they are replicated to all the replicas" but still they consume only from leader replicas
1. It will restart and start pulling messages from it's previous offset. 2. The consumer must wait for the write to be replicated across all partitions (that's the high water mark). It will not find the new messages till they have been replicated.
This was such an insightful video! I really enjoyed the part about kafka . It reminded me of a video series I recently made on S3 buckets, where I dive deeper into core concept of S3 buckets like replication,storage tiers. If anyone’s interested, feel free to check it out - I’d love to hear your thoughts!
Thank you for watching.
Also, NEW EPISODE: interviewready.io/learn/system-design-course/building-an-ecommerce-app-1-to-1m/7-the-case-of-the-expensive-cache
Wow I was just out of college when I first found your channel.
I’m a fresh senior now preparing for interviews, great video!
Glad to read this, cheers!
Have been following you since long and i am so impressed with the consistency and humility. Yoir explanations are simple and you seem like a college mate we are getting to learn from.thanks for all the free content
Cheers!
Such a simple yet elegant video Gaurav!
Great Explanation with lovely animation !Thanks Gaurav Bhai !
Been trying to study this for the best to weeks, and this is by far the best youtube on the topic
Thank you!
zookeeper is no longer used in the latest version of kafka. The newer version is Kraft using consensus protocol among brokers. Makes it more scalable
Yes, they use RAFT now 😁
way better than zookeeper for managing live clusters...
It’s a great video. It covered the features of kafka well. I feel the way you explain a problem and explain how to overcome it might not be obvious to the listeners that kafka automatically takes care of these.
For eg, you said multiple consumers might consume a message from the same influencer topic leading to notification duplication, and you explained consumer grouping as the solution to it. But you might have missed to explain that it is not actually possible, any extra consumers more than one active on a partition sits idle and kafka takes care of it automatically. Viewers might think that as some inherent problem with kafka and they would be required to handle it. Apart from these minor things the video is very good. Thanks for making it!
Thank you!
I have worked on sqs n sns but this helped me understand that concept
Thanks a lot 🎉🎉
Cheers!
I haven’t received my Amazon delivery yet. What might be the possible reason?
@@Engineerhoon someone in Amazon must have missed a # in Amazon code
Sorry are you sure you are on the right video?
@@gkcs kafka must have lost the order event
@@gkcs haha
A missing #
Tomorrow is exam and this video was saviour for the kafka topic
Awesome! The best kafka internal architecture video
Man thanks for sharing this
awsm explanation bhaiya!... understood every bit!
Cheers!
Amazing video as always! Really informative!
Thank you!
6:58 If I remember correctly, ZooKeeper uses the Zab (ZooKeeper Atomic Broadcast) protocol instead of Paxos for consensus. Starting with version 2.8, Kafka no longer relies on ZooKeeper for leader election. Instead, it uses KRaft (Kafka Raft), Kafka’s implementation of the Raft consensus algorithm, for managing metadata and coordination internally.
Intresting bro looking forward to work in Kafka... Soon i am going to complete my HTML class 🎉
Hi @gkcs, it looks Kafka partition leader handles both read & write.
Write Acknowledgments: The level of acknowledgment (acks) from the leader to the producer depends on the producer configuration:
acks=0: The producer does not wait for acknowledgment.
acks=1: The producer waits for acknowledgment only from the leader.
acks=all: The producer waits for acknowledgment from all in-sync replicas (ISR).
[ Read ] Consistency: For strict consistency, consumers can be configured to only read from committed offsets (data replicated to all in-sync replicas).
Also newer version of Kafka (2.8 onwards) introduced KRaft (Kafka Raft) as replacement for Zookeeper to simplify operations & system design.
Ah, lots of improvements over the 2011 paper I see. Thank you for sharing them!
you are amazing brother!
Very well explained
very useful , Thanks Gaurav
Great video 🤜🤛
Thank you :D
Thanks again for a great video.
Thank you!
Nice explanation.
Thanks!
You are welcome!
1) Why do you call yourself gkcs
2) These new videos you upload, do you add them to your system design course as well? (I have enrolled and I am trying to figure out if your course is enough, or if I need to watch the videos as well)
Great video as always btw 😁
1. It stands for General Knowledge Counter Strike. It was my cs handle in college, and it stuck with the RUclips channel :D
2. We typically go into more depth at InterviewReady with the research papers. We dive into the papers as a group during the live zoom sessions. You can see the recordings here:
interviewready.io/learn/system-design-course/scaling-memcached-at-facebook/november---memcached-reading
Thank you!
@@gkcsthank you for taking the time to respond :) I'll surely check it out, all the best!!
I asked the first same question years ago, I initially thought it was General Knowledge Computer Science.
@gkcs put up a poll after that If I remember correctly, and then I got to know the real full form
At 5:36, why didn't the message no. 4 get replicated to the other partitions after it was received?? That way, when middle kafka broker would receive 5, everything would remain consistent
that was just a problem scenario for explaining possible data inconsistency which doesn't happen ...
if you set acks to leader or min sync replicas to 1 this can still happen
Great thanks Gaurav Sen
thankyou absolute amazing
I can use redis as a quque, also there is rabbitmq, why is it better to use kafka?
Thanks Gaurav!
You said kafka uses paxos, but isn't zookeeper atomic broadcast used ?
The timing of this video is like it was made for me :P
Cheers!
This is gold...
Thank you!
Well explained, I’m a Frontend developer and heard people talking about Kafka so this helps me to be part of discussion. Also, can you complete the Application cache box at 13:18. It triggered my OCD 😂
Hahhaha 😁
I see you have mentioned multiple time that kafka consumer reads from replicas. But I guess this is incorrect, both producer and consumer only interact with leader at any point of time. Can you confirm this?
You are right where you mentioned "Consumer consume message only once they are replicated to all the replicas" but still they consume only from leader replicas
can you make videos in dark mode?, it's too bright🐱, the board and animation also white?
Is it true that zero copy does not work when SSL/TLS encryption is enabled?
stackoverflow.com/questions/58651562/how-to-best-address-the-adverse-impact-of-ssl-encryption-on-zero-copy-optimizati
1. What if consumer itself crashes?
2. Will the consumer wait for the write to be replicated across all replicas of the partition?
1. It will restart and start pulling messages from it's previous offset.
2. The consumer must wait for the write to be replicated across all partitions (that's the high water mark). It will not find the new messages till they have been replicated.
Is the kafka replicas internally that way or the user need to configure it as a system design?
They are that way, and can be configured by the admin.
Why don’t you continue to finish the Kafka course?
I have a course at InterviewReady on System Design. Do check it out :D
We can also use rabbitMq
This was such an insightful video! I really enjoyed the part about kafka . It reminded me of a video series I recently made on S3 buckets, where I dive deeper into core concept of S3 buckets like replication,storage tiers. If anyone’s interested, feel free to check it out - I’d love to hear your thoughts!
Thank you!
@@gkcs Thank you sir , I have learned so much from you.
trying out to make my own niche
is it same like rabbitmq and redis? what is the difference?
@@TheHardikupadhyay87 rabit mq is kind of same but it has different qualities and use cases
redis is used for cache and not as message queue
@@TheHardikupadhyay87 Yes but it’s pull based and not push based like rmq.
Zookeeper works like eureka server?
Have a look at the third point here: github.com/Netflix/eureka/wiki/FAQ
15:05 money money money 🤑
I can feel the cash through the screen :P
Present Sir
:D
Videos topic is great but i feel there is not a very good explanation to a fresher who watch it very first time
what were you doing before speaking for the video ?! 😂
Exactly, I had to pause the video to make sure it wasn't me having a sudden heartbeat rise.😂
I was staring creepily at the camera.
@@gkcs some design flaw in the system 😂😂
Gourav Sen is captioned as god of sin 😂
you seem like a fan of Suniel Shetty
Your name was spelled as God of Sin in the transcript 😂😂
Now it's turned into an Easter Egg :P