Everything you need to know about Kafka (With Demo) to work as a PRO | consumer groups | partitions

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
A consumer group is a set of consumers which cooperate to consume data from some topics. The partitions of all the topics are divided among the consumers in the group.
Partitioning takes the single topic log and breaks it into multiple logs, each of which can live on a separate node in the Kafka cluster. This way, the work of storing messages, writing new messages, and processing existing messages can be split among many nodes in the cluster.
This video DEEP DIVE in to those area and discuss how should configure based on use cases.
To support the channel : paypal.me/helloKrish
Facebook: / krish.dinesh.official
Instagram : / krish.dinesh.official
twitter: / krishantha (@krishantha )

[@siamak.hatami]

Thanks a lot.

[@chamithkarunakalage6831]

pretty clear. short and sweet

[@KangoV]

Awesome. Very well explained. Subscribed.

[@nuwanrathnayake1712]

Well explained!!!

[@GaweshPrabhashwara]

thank you krish!

[@user-gn7du2nl5n]

Can't you do a small project combining all these stuff (Microservices, Kafka, RabbitMQ, Redis, etc.)? so that it will be very clear how to use this knowledge in a real world project. Thank you.

[@shubhambansal5487]

Can you please share the code repository link that you have used in this video ?

[@vickygoyal7987]

@Krish What to do, if producers are producing at faster rate however consumers are consuming at slower rate?

[@krish]

add more partitions and increase number of consumers. other wise messages will be “lag”

[@SanjeevKulkarniWorld]

How to achieve order of messages across partitions when having multiple consumers in a group? If we use the key, will be able to manage the order? Also if i use topic partition can we achieve this? Also want to understand on ack handling and manual retry

[@krish]

you can do but its very complicated. ill try to cover. but design to stay away from that. regarding manual retry will be other video soon

[@SanjeevKulkarniWorld]

@@krish Thank you for putting your time and enbling others 👍

[@ranjanasamaranayake1006]

Is same partition is not attached to more than one consumer in same consumer group because of avoiding duplication?

[@krish]

correct. that is one reason. also can support scaling

[@sundhararulmani5355]

May i know the kafka docker image name?

[@krish]

docker compose is in my github