Check out my fundamentals of Database Engineering udemy course with over 20 hours of deep database systems lectures. Head to database.husseinnasser.com for a discount coupon. Link redirects to udemy with coupon applied.
27:28 answer: Every broker on a Kafka cluster is also named a Bootstrap Server. All brokers have the metadata required for the clients (producer or consumer) to discover brokers. When a client connects to one of the brokers (which are already configured as bootstrap servers in the Kafka configuration) it makes a "metadata request". The response includes information about topics, partitions, leader brokers, etc. Once the client gets this info, then - in the case of a producer- it makes the write request directly to the leader broker for that specific partition.
I always got demotivated by the complexity I experienced when I tried to learn this on my own but this one is a gem. I know it might have taken a lot of learning and experience for you to explains this in a simple manner Nevertheless thank you for sharing your knowledge!
i had to deal with Kafka in one of my project, so searched for some tutorials, As soon as I saw that you have made a video on this, i had a sense of relief that i dont have to worry about the resource atleast. Thanks for all your efforts man.
I'm lucky enough i found your channel at the start. I am currently a beginner on kafka, but bro, you're good at making things simple and explaining Thanks so much
Sweeeeet... did not feel like a tutorial, but like a friendly chat between deeply connected friends, every word makes clear sense. Kept me engaged every second till the end, yes I made to the end. Well done Sire... 👏👏👏
I just wanted to express my heartfelt appreciation for the amazing Kafka crash course, Your dedication to simplifying complex topics shines through, and I can't thank you enough for making Kafka so much easier to understand.
Great tutorial as always. Literally a crash course fit into about an hour. Has everything to get started with kafka, especially because of the hands on.
bro your tone when explaining these videos are so good. Didn't even feel like someone was teacahing me. It was as if a friend was talking to me lol. Good job man. Your excitement is intoxicating. :)
Course Table of Content time codes: (EDIT you should also see youtube chapters) Download slides here payhip.com/b/Y0Cq Kafka Broker: 2:32 Kafka Producer: 5:39 Kafka Consumer : 7:11 Kafka Partitions : 8:34 Queue vs Pub/Sub 12:50 Consumer Group 16:47 ZooKeeper 23:06 Example 29:45 Kafka Pros & Cons 1:02:48 Summary: 1:17:00 Example time codes Spin up Zookeeper 31:37 SPin up Kafka 34:00 Connect to Kafka (40:30) Create Kafka Topic (45:30) Create a Producer (49:44) Create a Consumer (55:24) Final test (1:00:00)
I just watched your RabbitMQ crash course and at the end of the video you mention how Kafka better at some point, then I bumped into this great video. Very clear explanation, love your video!
Luqman S thanks Luqman! Each software its use cases, I think RabbitMQ Is trying to be more than it really is (a message queue) where as kafka was designed to be a pub/sub and a message queue. If your app needs a queue and a pub/sub (or maybe streaming) than Kafka . If you just need a queue RabbitMQ is the way to go and kafka will be an overkill I think.. I would even consider zeromq as a lightweight mq.. Good luck!
Not only the content, but the way you say it, and your enthusiasm, is something else. You sir deserve all the best, because you are, with no doubt, in the better's shelf..
this is the only tutorial i found that go in the depth and details of a pro should know , and i can compare Kafka with other messaging system like JMS and MQ. Thanks so much for sharing👍👏🙏
I didn't know you before this video, but after this video? I'm gonna watch everything you upload man. Every tutorial you're selling? I'm buying! Obviously I liked & subscribed.
Made it to the enddddd ... I know how kafka works “at least before witching this tutorial “ but after this tutorial many things become clear to me ... Thanks a lot
Thanks, I feel like I learned everything I need to know about Apache Kafka from this video. Sure, there are more advanced concepts. But this is the essence.
I made it to the end. Excellent. I don’t remember which video previously I saw was so long MD not boring at all. In fact you made it so interesting. May Allah (God) guide and bless you.
Hussein, You are a legend!!! Any one can be a master if they follow to your conversations. I like the way yo have a full conversation with yourself - such a good dialogue asking and answering yourself
@@hnasr One question though.. Shouldn`t the consumer allways get all messages in the topic (within its partition)? I can see that the property "fromBeginning" i set to true, but when i restart the consumer, it still waits for new mesages. It does not replay the old ones?
At 39:32 this doesn't work for me. I try to connect Kafka to Zookeeper but I get the following error: [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: localhost/127.0.0.1:2181: Connection refused I'm using Windows, I used localhost instead of machine name on the environment variable
@@yashpalgandhi4108 Use the following command to get the internal IP of the zookeeper docker inspect zookeeper --format='{{ .NetworkSettings.IPAddress }}'
Answer to - 27:25 - The producer sends a Metadata request with a list of topics to one of the brokers in the broker-list you supplied when configuring the producer. The broker responds with a list of partitions in those topics and the leader for each partition. The producer caches this information and knows where to redirect its produce messages. In case of failure while producing, failed broker's data (topics and its partitions) dynamically linked to existing replica which is present on another broker via topic's replication and new leader's information is communicated to the client (producer).
Appreciate your efforts...like the way you clearly say what you know/understand & what you dont...else I see most people in Industry are faking the intelligence & understanding about any technology & they pretend it so perfectly like they themselves have developed that tech...while those who actually know/understand the tech properly are very few & rare... So always good to watch your videos to understand something in simple words rather mugging those funky & techie words.
Hi Hussein, I love your way of explaining complex topics, kudos for good work. For some time, I was also confused about how the producer sends to the leader only, then I found out this in Kafka docs "The producer sends data directly to the broker that is the leader for the partition without any intervening routing tier. To help the producer do this all Kafka nodes can answer a request for metadata about which servers are alive and where the leaders for the partitions of a topic are at any given time to allow the producer to appropriately direct its requests." So I think the producer query the info before writing.
@KuldeepYadava - Thanks for answering the question. On top of this I have question does the producer request for "give me the leader" every time before sending message or it keeps doing on certain interval.
Amazing explanation Hussein. Every single bit of your lecture is fully understandable. You made difficult thing easy with practical example. You rocked! Good Luck.
Amazing tutorial! Really find the topic interesting and easy to understand broker, producer, consumer etc.. Thanks for coming up with a great video. Hope we will get to see more on this.
Hey ! Good video that I enjoyed it a lot. I have heard of kafka and wondering what is for and how to use, and now everything is clear for me thanks to your simple explication. and I hope you will make more videos in the future about full stack developing.
Check out my fundamentals of Database Engineering udemy course with over 20 hours of deep database systems lectures. Head to database.husseinnasser.com for a discount coupon. Link redirects to udemy with coupon applied.
I love it and the way it’s presented! Will announce on LinkedIn when I get my cert. Thanjs for putting this together!
I think you should update this video as Zookeeper is no longer needed for Kafka.
27:28 answer:
Every broker on a Kafka cluster is also named a Bootstrap Server. All brokers have the metadata required for the clients (producer or consumer) to discover brokers. When a client connects to one of the brokers (which are already configured as bootstrap servers in the Kafka configuration) it makes a "metadata request". The response includes information about topics, partitions, leader brokers, etc. Once the client gets this info, then - in the case of a producer- it makes the write request directly to the leader broker for that specific partition.
Thanks for the explanation
This should be pinned. Thank you!
if all the brokers maintain this metadata then what's the role of zookeeper.
@@HarshJain-lk8cw The use of zookeeper is that only one leader is there for a partition at any given time by using some kind of lock registry.
This should be pinned. Thanks!
I rarely comment on videos but this was genuinely one of the best computer science tutorials I've ever seen!
Thank you Anish for taking the time to write this appreciate you!
I always got demotivated by the complexity I experienced when I tried to learn this on my own but this one is a gem. I know it might have taken a lot of learning and experience for you to explains this in a simple manner Nevertheless thank you for sharing your knowledge!
Thank you Jithin! And Remember to ask in the community to get access to the slides if you would like to since you are a member❤️
i had to deal with Kafka in one of my project, so searched for some tutorials, As soon as I saw that you have made a video on this, i had a sense of relief that i dont have to worry about the resource atleast. Thanks for all your efforts man.
I'm lucky enough i found your channel at the start. I am currently a beginner on kafka, but bro, you're good at making things simple and explaining
Thanks so much
Sweeeeet... did not feel like a tutorial, but like a friendly chat between deeply connected friends, every word makes clear sense. Kept me engaged every second till the end, yes I made to the end. Well done Sire... 👏👏👏
I just wanted to express my heartfelt appreciation for the amazing Kafka crash course, Your dedication to simplifying complex topics shines through, and I can't thank you enough for making Kafka so much easier to understand.
Great tutorial as always. Literally a crash course fit into about an hour. Has everything to get started with kafka, especially because of the hands on.
bro your tone when explaining these videos are so good. Didn't even feel like someone was teacahing me. It was as if a friend was talking to me lol. Good job man. Your excitement is intoxicating. :)
how do you only have 23k subs?? you're so good at explaining difficult concepts, tysm!
61k
@@ballonura 79k
86.6K
@@mazharuddin3647 88.3k
@@johncerpa3782 88.4k
Course Table of Content time codes: (EDIT you should also see youtube chapters)
Download slides here
payhip.com/b/Y0Cq
Kafka Broker: 2:32
Kafka Producer: 5:39
Kafka Consumer : 7:11
Kafka Partitions : 8:34
Queue vs Pub/Sub 12:50
Consumer Group 16:47
ZooKeeper 23:06
Example 29:45
Kafka Pros & Cons 1:02:48
Summary: 1:17:00
Example time codes
Spin up Zookeeper 31:37
SPin up Kafka 34:00
Connect to Kafka (40:30)
Create Kafka Topic (45:30)
Create a Producer (49:44)
Create a Consumer (55:24)
Final test (1:00:00)
Dude, I made it to the End. You kept us engaged throughout. I have soo much to learn about Comp Sci!
I just watched your RabbitMQ crash course and at the end of the video you mention how Kafka better at some point, then I bumped into this great video. Very clear explanation, love your video!
Luqman S thanks Luqman! Each software its use cases, I think RabbitMQ Is trying to be more than it really is (a message queue) where as kafka was designed to be a pub/sub and a message queue. If your app needs a queue and a pub/sub (or maybe streaming) than Kafka . If you just need a queue RabbitMQ is the way to go and kafka will be an overkill I think..
I would even consider zeromq as a lightweight mq..
Good luck!
Another perfect tutorial. Everything is explained very nicely. Thanks for making Kafka easy to understand in this tutorial.
Atif Ali thanks Atif for your support! Happy I could make such complex topic easy to absorb.. still I didnt cover all what kafka has to offer! Cheers
This was an awesome tutorial. You're incredibly good at explaining things and keeping people engaged when consuming so much information.
One of the best tutorials i’ve seen. Great class and very funny. Subscribed and waiting for more content!
Super video! I applauded for $2.00 👏
❤️❤️❤️❤️
amazing tutorial, can't have words to express, you make it so simple to understand these concepts. huge respect.
Couldn't learn kafka better in such a short time. Thanks Hussein.
Today i understood, its not the subject its the way of teaching...thanks for sharing this wonderful knowledge
Not only the content, but the way you say it, and your enthusiasm, is something else. You sir deserve all the best, because you are, with no doubt, in the better's shelf..
this is the only tutorial i found that go in the depth and details of a pro should know , and i can compare Kafka with other messaging system like JMS and MQ. Thanks so much for sharing👍👏🙏
You explain by answering all the questions people get when learning a topic. Most tutorials just explain concepts.
professionalism + enthusiasm + enjoyment + wonderful content= Hessein Nasser
John Shaheen you make me blush John 😊 thanks !!
I made it to the end. :P Thank you so much for the amazing content Hussein.
Super video! I applauded for $5.00 👏👏
❤️🙏🙏
I didn't know you before this video, but after this video? I'm gonna watch everything you upload man. Every tutorial you're selling? I'm buying! Obviously I liked & subscribed.
This is the BEST educational channel!!! Bravo!
I made it to the End! Thank you for that video, I can now say I understand Kafka a lot better than 1 hour ago :)
Made it to the enddddd ... I know how kafka works “at least before witching this tutorial “ but after this tutorial many things become clear to me ... Thanks a lot
❤️❤️❤️ روعه!
@@hnasr هههههههههههه اسعدتني في بداية يومي
Thanks, I feel like I learned everything I need to know about Apache Kafka from this video. Sure, there are more advanced concepts. But this is the essence.
I made it to the end. Followed every step and it worked nicely. Thank you for the effort... really helpful
I made it to the end. Excellent. I don’t remember which video previously I saw was so long MD not boring at all. In fact you made it so interesting. May Allah (God) guide and bless you.
Hussein, You are a legend!!! Any one can be a master if they follow to your conversations. I like the way yo have a full conversation with yourself - such a good dialogue asking and answering yourself
Thank you! I am happy you noticed the conversation style, i have been working on it 😊
By far the best introduction video to Kafka I have seen. Great examples and nice build up to demos that puts theory into practice.
Hans-Jacob Melby thanks Hans!
@@hnasr One question though.. Shouldn`t the consumer allways get all messages in the topic (within its partition)? I can see that the property "fromBeginning" i set to true, but when i restart the consumer, it still waits for new mesages. It does not replay the old ones?
made it to end. Amazing video. you made Kafka look so simple, and the best part-> you showed a code demo. clap. clap.
Excellent practical explanation of Kafka, I wish every technical tutorial was as easy as this. Thumbs up!
Great video, thanks for making it. Love your accent! You're very easy to understand, follow, and your excitement is fantastic!
vbrtrmn glad to know thank you! Lots of people have trouble with my accent glad you like it!
You really know how to do tutorials, great video!
Made it to the end, now gonna implement the backend of a texting app using Kafka
Best explanation of Kafka on the internet!
Made it to the end! Awesome video Hussein :)
Man, you are the best, thanks a lot for creating such valuable content for free
Made it to the end!! Amazing content!
The explanation and illustration is clear and concise! Thanks for the comprehensive tutorial!
made it to the end , noice, hopefully i consistently get better at backend and devops :3
Thank you for putting in the effort to make a short and clear to understand video.
The best stuff on kafka . Made it so simple
I made it to the end. Love the long videos and the content 👌👌👌
sidra tariq thanks Sidra I like long format too 😊🙏
At 39:32 this doesn't work for me. I try to connect Kafka to Zookeeper but I get the following error: [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: localhost/127.0.0.1:2181: Connection refused
I'm using Windows, I used localhost instead of machine name on the environment variable
same issue here as well
@@yashpalgandhi4108 Use the following command to get the internal IP of the zookeeper
docker inspect zookeeper --format='{{ .NetworkSettings.IPAddress }}'
Unable to canonicalize address MacBook-Pro:2181 because it's not resolvable - what am i doing wrong bros? This happens when im firing up kafka.
Answer to - 27:25 - The producer sends a Metadata request with a list of topics to one of the brokers in the broker-list you supplied when configuring the producer. The broker responds with a list of partitions in those topics and the leader for each partition. The producer caches this information and knows where to redirect its produce messages. In case of failure while producing, failed broker's data (topics and its partitions) dynamically linked to existing replica which is present on another broker via topic's replication and new leader's information is communicated to the client (producer).
Appreciate your efforts...like the way you clearly say what you know/understand & what you dont...else I see most people in Industry are faking the intelligence & understanding about any technology & they pretend it so perfectly like they themselves have developed that tech...while those who actually know/understand the tech properly are very few & rare...
So always good to watch your videos to understand something in simple words rather mugging those funky & techie words.
Thanks a lot, Husseine. You definitely demystified Kafka.
I was looking for your channel , you really good at explaining things you helped me once , I'm here again , thank you andn jazkAlllah khayran .
This course is great because of the example. It's much better than some videos in which only some introductions are represented.
Amazing, Hussein! Thanks for the overview of Kafka. I've always been a bit afraid of this beast, but now I understand at least the basics
oh my god I love this video so much =]] this is such an amazing tutorial and how funny the instructor is : D
made it to the end, and every detail was fully explained thanks.
Finally a funny geek! I have enjoyed the video
Thank you for this, don't know how I missed it
This is THE BEST video on this topic. Thanks a lot!
Hi Hussein, I love your way of explaining complex topics, kudos for good work.
For some time, I was also confused about how the producer sends to the leader only, then I found out this in Kafka docs
"The producer sends data directly to the broker that is the leader for the partition without any intervening routing tier. To help the producer do this all Kafka nodes can answer a request for metadata about which servers are alive and where the leaders for the partitions of a topic are at any given time to allow the producer to appropriately direct its requests."
So I think the producer query the info before writing.
@KuldeepYadava -
Thanks for answering the question.
On top of this I have question does the producer request for "give me the leader" every time before sending message or it keeps doing on certain interval.
I made it to the End!!! Thanks so much Hussein!
made it to the end 🤟. It was a very good explanation with great clarity. Really Appreciate your work! Thanks a lot 🙏
This is brilliant and concise. Been looking for a channel like this 👍🏾
M Zaman enjoy the content 🙏
Clear and easy to understand. Although it's a long video, but not boring at all.
Made it to the end! Yay! Nice video Hussein!
This guy nailed it completely. Luckily, I landed here!!!
your contents are more enjoyable then any Phd holding professor! lol:))))))))
You're killing me man "LISTENERRRS" 🤣 ! Great video by the way.
Very nice tutorial. Got a clear understanding of Kafka, especially the part of queue, pub sub, very helpful! Thank you so much!
Amazing explanation Hussein. Every single bit of your lecture is fully understandable. You made difficult thing easy with practical example. You rocked! Good Luck.
Hi Hussein,
Really nice video.. And yes I completed till the end 'WITHOUT Sleeping 😃.
Such a nice tutorial. You earned a subscriber.. 😊
anup kumar thank you Anup 🙏 welcome to the channel glad you enjoyed the content
This is a great video to understand what Kafka is and how it works, very well explained!
Cant say techical corrrect, but this video clears a lot of common confusions associated with Kafka fundamentals.
Amazing tutorial! Really find the topic interesting and easy to understand broker, producer, consumer etc.. Thanks for coming up with a great video. Hope we will get to see more on this.
It's good that you included jordan speech...i was just googled that video and came back
Amazing video. A gifted teacher like this is quite rare ♥Thanks a lot!!
btw made it to the end 😆
Brother you're videos are phenomenal they are just top notch.
Really enjoyed this informative session and yeah, made it to the end !!!
Huge props for trying to understand things before you teach them!
Best content on Kafka for beginners 👌
Made it till the end !
You can explain something in a super simple manner if and only if you understand it
And you do, Thanks !
Watched this video in one session, also applied the practice on my machine, what a great content you are creating Hussien, a must rewatch as always 😉
Hey ! Good video that I enjoyed it a lot.
I have heard of kafka and wondering what is for and how to use, and now everything is clear for me thanks to your simple explication. and I hope you will make more videos in the future about full stack developing.
Glad to hear that!
Just started watching and already loving it. I wish you a happy new year and hoping to see more great content like this.
Your skillset is brilliant, amazing and fantastic. Thank you so much!
Amazing video Hussein ! The way you explain helps to register the information in our memory :-)
I appreciate the content. no boring part :)
I bet Borat learnt and got your voice :D. Jokes apart, really learnt and above all enjoyed your video. Keep it up. Subd! :)
Amazing video! You have the talent to explain the complicated stuff really clearly😊
best video on kafka. Thanks a ton!
You made it very simple to understand. A very complex topic.
This was actually good. I was able to get it to run January 2025.
the simplest and best tutorial ever!! Good Job
Love the Gary Vee references😄 awesome video
Thank you. Best video so far for learning kafka. 👍
Amazing video, you have made a complex topic extremely simple with your style of teaching.
truly amazing-so simple and incredibly useful. Great job!