Thank you KLNP1978! If you want to see the lab videos on Consumer Group, then you can refer these 2 videos -- Kafka Consumer Groups Demo using CLI| Kafka-Python ruclips.net/video/yKzdum_vX7k/видео.html Kafka Consumer Group Demo using Python ruclips.net/video/XRRyO6b9Wqg/видео.html Hope these will be helpful! Happy Learning
hi, a question: how do we load balance consumers in a pub sub system? for example if we take example of online chat, millions of users(consumers) exchange message from each other( let's say private message) and imagine all these consumers are listening on one topic(call it "messages"). since each and every consumer needs to get the message(from all the partitions) we can only assign unique group id to each user(consumer).. if the rate of message publishing is too heavy obviously one consumer cannot hold the load (since each consumer listens from multiple partitions and millions of messages)... so, is there a way if we can take off this load on one consumer ? One approach I thought was if we can assign one topic per user (eg: alice_topic, bob_topic) then each consumer doesn't need to listen for all the messages on a topic irrelevant to that user. User can just listen on his channel and any other publisher can just post to opponent user... Is this an optional solution since we end up in creating millions of topics(one per user ).
Great job !
Thank you KLNP1978! If you want to see the lab videos on Consumer Group, then you can refer these 2 videos --
Kafka Consumer Groups Demo using CLI| Kafka-Python
ruclips.net/video/yKzdum_vX7k/видео.html
Kafka Consumer Group Demo using Python
ruclips.net/video/XRRyO6b9Wqg/видео.html
Hope these will be helpful! Happy Learning
hi, a question: how do we load balance consumers in a pub sub system?
for example if we take example of online chat, millions of users(consumers) exchange message from each other( let's say private message) and imagine all these consumers are listening on one topic(call it "messages").
since each and every consumer needs to get the message(from all the partitions) we can only assign unique group id to each user(consumer)..
if the rate of message publishing is too heavy obviously one consumer cannot hold the load (since each consumer listens from multiple partitions and millions of messages)... so, is there a way if we can take off this load on one consumer ?
One approach I thought was if we can assign one topic per user (eg: alice_topic, bob_topic) then each consumer doesn't need to listen for all the messages on a topic irrelevant to that user. User can just listen on his channel and any other publisher can just post to opponent user... Is this an optional solution since we end up in creating millions of topics(one per user ).