Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/
Love your dedication to educate the community. Creating such presentations that you use to explain consumes time, really appreciate your hardwork and time.
This is the 17th video in this playlist and I still wonder the way you explain each concept. All topics were very clear and the pictorial representation was amazing.
Hi,Thank you sir for sharing your knowledge.The way u take the concept and explaining is awesome. After watching your spark videos i got the confidence that i can face interviews with more confidence .Thanks for showing way how to do in real time using GCP. Now i started learning kafka parallel with spark. These videos are in terminal.Can you please explain kafka also in real time with GCP. That will be a great advantage for those who are moving to this ecosystem. Thank you
Thanks for good tutorial. Have a question around authentication. How broker authenticate consumers if it is valid? What if someone know the configuration and create his own consumer to pull the data?
1) 10k producers sending the 50 records(one record at a time) of data for each 3min, can you please suggest me that how many partitions, consumers do i need? 2) What will happen if consumer count is more than Partition count, whether consumers consumes duplicate data( consumes more than once)? 3) what is the RAM consumption, ROM usage? 4) How much time consumers will take to read all the records?
1) You can decide the number of consumers based on some experiments. I suggest a small POC with the real records and then do some maths. 2) No, additional Consumers will not read any partitions. 3) Memory consumption depends upon your message size, and the number of in-flight messages is the main factors. 4) Depends on many factors. The size of the record and network speed are the major ones.
Thanks again for a good tutorial series, At the time of executing consumer.poll(), does it fetches all available messages in the assigned partition at once ? Is there any control to specify how much data(at most) the consumer is interested in receiving in a single poll ? On a related note, suppose that my consumer received 10 messages on a single poll() but failed(consumer died) while processing the 5th message. Is there a way for the broker to know that it has to re-deliver the messages to the newly re-assigned consumer? Appreciate if you could give some insights on such scenarios. Thanks.
Question- Only group name is created and no assignment of how many consumers should be created for a group in order to balance for multiple partitions. Kafka decides numbers of consumers automatically?
No, Kafka doesn't decide the number of consumers. We have to start Consumers, as many as we want. Just keep the Group Name same for all consumers, and they will share the workload.
Hi sir, i did code same but i am seeing logs updating continuously along with producer messages. I mean when i run consumer its displaying logs continuously without stopping. The logs are like fetching data or uncommitted offsets etc. Please help me why it is coming..
What happens if there's 5 partitions, and I've created 2 consumer groups, both having 5 consumers? How does the GROUP CO-ORDINATOR balance in such a case? Does it randomly assign the 5 partitions among these 10 consumers? Another question I have is on the message itself -- If a message is too big, is the message split and pushed to different partitions ? or same partition ? [ I'm sorry I couldn't post this in one of the producer related videos ]
Q1- Each Group will read all 5 partitions. Since there are 5 consumers in each group, they get one partition each. Q2- If the message is too big, you get an exception. It can't be sent.
Hello! Thanks for the great tutorial. How can I find the number of messages in a topic using Java and is there is a way to check if a particular message is in a topic? Please share your thoughts. Thanks!
conumer reads all records from topic higher than current offset or particular number of records? ,if its a particular no of records ,how consumer decides how many records to read from topic can u please explain
However, Do you connect to a Leader? No, Right. When you create a consumer, You just connect to the cluster by supplying any broker address. So, as a consumer, you never know who a leader is? If one Leader goes down, a new leader is elected.
Saying there are 5 partitions in a topic and 4 consumers in a group to consume the topic messages. Now I start a independent new consumer to consume the topic messages, how does the group coordinator to handle this case?
What do you mean by the independent consumer? If it is part of the new group, it consumes all partitions. If it is not part of any group, then you have no group coordinator assigned.
Ok, That's a consumer who is not part of any group, so it has to assign topic and partition to itself using "assign" API. It should take care of everything that a Kafka group management offers. There is an example for such a consumer in "Exactly once processing" video.
Create a consumer and also a producer, both in same application and you can do it. Join My Udemy course and I am adding an example in the same. It is free for few more days until I finish the full content.
Sir i have a question is it possible to delete a message from topic once a subscriber have received a message something like subscriber will send an acknowledgement to producer or topic and then message can be deleted.
What if I do this: props.put("bootstrap.servers", someIpAddress + ":9092"); ? Shouldn't my consumer receive messages from a functioning producer from a host having the IP address = someIpAddress? Is there any problem if many consumers bind to the same port of the producer (i.e. someIpAddress:9092)? For instance I have the following situation: producer on a host, consumer1 on the same host, consumer2 on a different host. Consumer1 receives the messages sent by the producer, consumer2 DOES NOT... the code is identical, I only changed "localhost" with the IP address of the producer (for consumer2).
I am not sure if I follow your question. However, I read a fundamental misconception. The consumer never directly interact with a producer. There is a Broker in between. The bootstrap server is the Broker address, not the producer. Hope this helps.
If the consumer is on a different machine than the one of the broker and if i do this in the consumer: props.put("bootstrap.servers", "123.12.12.12:9092"); where 123.12.12.12 is the address of the broker, shouldn't the consumer receive the messages?
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
www.learningjournal.guru/courses/
Love your dedication to educate the community. Creating such presentations that you use to explain consumes time, really appreciate your hardwork and time.
Best Kafka tutorial I have seen. Thank you for the wonderful, neat and clear lecture
Thanks a lot Vasanth for your feedback. Appreciate you took the time to write back.
This is the best kafka tutoriel ever !!! Thank you buddy! we love u !
This is the 17th video in this playlist and I still wonder the way you explain each concept. All topics were very clear and the pictorial representation was amazing.
Such excellent tutorial. Your awesome sir!. The way you explain the concepts i liked it very much. It crisp and clear. Thanks!
great explanation Sir, Thank you😊
You are best trainer sir even seen as of now
Hi,Thank you sir for sharing your knowledge.The way u take the concept and explaining is awesome. After watching your spark videos i got the confidence that i can face interviews with more confidence .Thanks for showing way how to do in real time using GCP.
Now i started learning kafka parallel with spark. These videos are in terminal.Can you please explain kafka also in real time with GCP. That will be a great advantage for those who are moving to this ecosystem. Thank you
Excellent video!!!!. Explanation was very clear. Looking forward for more videos.
awesome tutorial, explained very well and clear all doubts
Amazing tutorial. Really like the way you explain each and every concept.
Thank you
amazing sessions to get ur basics right
explained in simple and good way..
Thanks for good tutorial. Have a question around authentication. How broker authenticate consumers if it is valid? What if someone know the configuration and create his own consumer to pull the data?
Short and sweet!
very good informative tutorial on kafka
Thanks
1) 10k producers sending the 50 records(one record at a time) of data for each 3min, can you please suggest me that how many partitions, consumers do i need?
2) What will happen if consumer count is more than Partition count, whether consumers consumes duplicate data( consumes more than once)?
3) what is the RAM consumption, ROM usage?
4) How much time consumers will take to read all the records?
1) You can decide the number of consumers based on some experiments. I suggest a small POC with the real records and then do some maths.
2) No, additional Consumers will not read any partitions.
3) Memory consumption depends upon your message size, and the number of in-flight messages is the main factors.
4) Depends on many factors. The size of the record and network speed are the major ones.
Thank you
Excellent way of explanation and Waiting for your Hadoop and hive tutorial as well
Thanks, It's coming soon, Before the Bahubali 2 :-)
lol :)
very informative .
good explanation
How to send data from Kafka consumer in spring boot to Victoria metrics.
Thank you sir...
Can you describe a clean way of closing the consumer mine keep stuck at the infinite loop
Thanks again for a good tutorial series, At the time of executing consumer.poll(), does it fetches all available messages in the assigned partition at once ? Is there any control to specify how much data(at most) the consumer is interested in receiving in a single poll ?
On a related note, suppose that my consumer received 10 messages on a single poll() but failed(consumer died) while processing the 5th message. Is there a way for the broker to know that it has to re-deliver the messages to the newly re-assigned consumer? Appreciate if you could give some insights on such scenarios.
Thanks.
Best Tutorials for Kafka.. If possible, will you able to post some videos about Mirrormaker?
Thanks. Your request for the new content is reasonable. I am adding it to my list for next Kafka series. I will start a new set in the near future.
Thanks:)
Question- Only group name is created and no assignment of how many consumers should be created for a group in order to balance for multiple partitions. Kafka decides numbers of consumers automatically?
No, Kafka doesn't decide the number of consumers. We have to start Consumers, as many as we want. Just keep the Group Name same for all consumers, and they will share the workload.
Thanks a lot. Understand.
how to get latest message enter into topic using java consumer
Hi sir, i did code same but i am seeing logs updating continuously along with producer messages. I mean when i run consumer its displaying logs continuously without stopping. The logs are like fetching data or uncommitted offsets etc. Please help me why it is coming..
Do you have video tutorial for Junit testing a kafka consumer?
Isn't we have to pass zookeper details in the consumer... rather passing the brokerlist?
Which API create coordinator and leader ?
Thanks sir...it is not possible consume and send m sg (s)from the same application for testing purpose
i want command which create consumers group
I tried to make 2 brokers in GCP. its throwing an error. It says only one boker available for replication faction
What happens if there's 5 partitions, and I've created 2 consumer groups, both having 5 consumers? How does the GROUP CO-ORDINATOR balance in such a case?
Does it randomly assign the 5 partitions among these 10 consumers?
Another question I have is on the message itself -- If a message is too big, is the message split and pushed to different partitions ? or same partition ? [ I'm sorry I couldn't post this in one of the producer related videos ]
Q1- Each Group will read all 5 partitions. Since there are 5 consumers in each group, they get one partition each.
Q2- If the message is too big, you get an exception. It can't be sent.
Hello!
Thanks for the great tutorial.
How can I find the number of messages in a topic using Java and is there is a way to check if a particular message is in a topic? Please share your thoughts. Thanks!
You can do all that however you must ask yourself why? Do you want to use a broker as a database to execute queries?
Sir, How can I provide a group name if I am starting a console consumer using the kafka script?
Sir how does the consumer get to know that it has finished consuming all records from the producer?Is there any technique for that?
I think you finish all videos and by the end of tutorial series, you should get the answer.
conumer reads all records from topic higher than current offset or particular number of records? ,if its a particular no of records ,how consumer decides how many records to read from topic can u please explain
It reads as much as it can within the specified time limit and other limits specified through configurations.
Question? what happens if there is a Leader broker fails from where consumer was consuming data?
A new leader is elected. I have a separate video on Fault tolerance in Kafka. Have you checked that? Hope that answers your doubt.
However, Do you connect to a Leader? No, Right.
When you create a consumer, You just connect to the cluster by supplying any broker address. So, as a consumer, you never know who a leader is? If one Leader goes down, a new leader is elected.
Could u Please help me. I have some dought . Please give a call on that number 9066898340
Saying there are 5 partitions in a topic and 4 consumers in a group to consume the topic messages. Now I start a independent new consumer to consume the topic messages, how does the group coordinator to handle this case?
What do you mean by the independent consumer? If it is part of the new group, it consumes all partitions. If it is not part of any group, then you have no group coordinator assigned.
you mentioned the concept of 'independentn consumer' at 2:20. I am confused what partition will the independent consumer works on.
Ok, That's a consumer who is not part of any group, so it has to assign topic and partition to itself using "assign" API. It should take care of everything that a Kafka group management offers. There is an example for such a consumer in "Exactly once processing" video.
Where do I need to place property file?, as we are not mentioning the property file path in the code
current directory.
can you help me how to copy data (say json format data) consumed from one topic to another topic after data being modified?
Create a consumer and also a producer, both in same application and you can do it.
Join My Udemy course and I am adding an example in the same. It is free for few more days until I finish the full content.
Hi Sir, here group_id is use if we have a consumer group. But is it required if we only use one consumer?
No. You dont need it.
Sir i have a question
is it possible to delete a message from topic once a subscriber have received a message something like subscriber will send an acknowledgement to producer or topic and then message can be deleted.
No. Subscriber and producer don't know each others. They always talk to broker. They never talk directly.
What if I do this: props.put("bootstrap.servers", someIpAddress + ":9092"); ?
Shouldn't my consumer receive messages from a functioning producer from a host having the IP address = someIpAddress?
Is there any problem if many consumers bind to the same port of the producer (i.e. someIpAddress:9092)?
For instance I have the following situation: producer on a host, consumer1 on the same host, consumer2 on a different host. Consumer1 receives the messages sent by the producer, consumer2 DOES NOT... the code is identical, I only changed "localhost" with the IP address of the producer (for consumer2).
I am not sure if I follow your question. However, I read a fundamental misconception. The consumer never directly interact with a producer. There is a Broker in between. The bootstrap server is the Broker address, not the producer. Hope this helps.
If the consumer is on a different machine than the one of the broker and if i do this in the consumer: props.put("bootstrap.servers", "123.12.12.12:9092"); where 123.12.12.12 is the address of the broker, shouldn't the consumer receive the messages?
Yes. That's the ideal case. You can't expect consumers to be executing on broker nodes.
.net ?
Hi, I have a question that, can we create kafka consumer for any mysql databases?
Yes, we can, But for that, we use Kafka Connect.
Thanks for your reply, are you talking about the "Confluent Jdbc" Connector??
Is there any Consumer API(in Java) Available for mysql as a consumer?
check kafka.apache.org/quickstart#quickstart_kafkaconnect
Thank you so much....I appreciate your Response.
Can you share the git hub link?
github.com/LearningJournal/ApacheKafkaTutorials
how can I download powerpoint of this course?
PowerPoint is not available. If you are looking for code to copy paste, it is available at the website.
No. I want to hold a seminar on Kafka in Iran.
So I wanted to get help from your good slides
Unfortunately, I haven't preserved those slides. You might want to take the screenshots of full-screen videos.
ok.thanks