Explained well. concepts are so clear. Below are the topics learned so far - 1. Verifying multiple partitions in the server logs directory 2. Checking the offset based on the flag parameter within the partition 3. Content check for each message within the partition
if someone is using kafka >2.4 , this approach of round robin(random partitioning) will NOT work, instead there is something called sticky partitioner where you'll see all msgs going to same partition. WHY? because we didn't specify keys in the start. The partitioning strategy by kafka has changed.
Hello Vignesh R , you can have a look in this video , I explained your scenario in detail -- ruclips.net/video/yKzdum_vX7k/видео.html Hope this will be helpful! Happy Learning
Hello Namrata Pathak, please post your doubt here , If I know that topic or have any idea , I will surely reply asap , also for any kafka related doubts , please check this playlist once , might help you if the topic is covered in the playlist already -- ruclips.net/p/PLjfRmoYoxpNrs0VmIq6mOTqXP52RfZdRf
Codes used in the demo :
--------------------------------------------
Start Zookeeper:
---------------------------
F:/kafka_2.12-3.2.0/bin/windows/zookeeper-server-start.bat F:/kafka_2.12-3.2.0/config/zookeeper.properties
Start Kafka-server:
-----------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server.properties
Create a topic:
-------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-topics.bat --create --topic demo_testing --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3
Start Producer:
--------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-producer.bat --topic demo_testing --bootstrap-server localhost:9092
Start Consumer:
----------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --from-beginning --bootstrap-server localhost:9092
Reading message from specific partition:
---------------------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --from-beginning --bootstrap-server localhost:9092 --partition 2
Reading message from specific offset inside a specific partition:
---------------------------------------------------------------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --bootstrap-server localhost:9092 --partition 2 --offset 1
Reading message from specific offset :
-------------------------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --bootstrap-server localhost:9092 --offset 0
Explained well. concepts are so clear.
Below are the topics learned so far -
1. Verifying multiple partitions in the server logs directory
2. Checking the offset based on the flag parameter within the partition
3. Content check for each message within the partition
absolute genius. amazing xplaination sir, thankyou so much.
Thanks and welcome
Wow, you explained the topics very well with practical as well as with internals. Its a very good tutorial for learning kafka. Thank you .
Thank you for your kind words Shobhit Arya! Happy Learning
Crisp explanation, 100% perfect
Thank you BigData Professional! Happy Learning
Great work man! Thank you
Thank you Kamal Nayan! Happy Learning
Thank you Sir
You are @user-vp4by6sb1m ! Happy Learning
if someone is using kafka >2.4 , this approach of round robin(random partitioning) will NOT work, instead there is something called sticky partitioner where you'll see all msgs going to same partition. WHY? because we didn't specify keys in the start. The partitioning strategy by kafka has changed.
You are correct Ayan , here are the videos where I have explained the same -- ruclips.net/video/-eo4IPUshRo/видео.htmlsi=o_tQF0TYGE5qFO_D
Thanks, I was also thinking about the same problem
Why does my data only save to partition 0 when I was like you, I tried adding many other input data examples, but it only jumps to partition 0
Please provide us the session for Confluent Kafka
One kafka producer,two consumer subscribes to kafka topic then how partition works?
Hello Vignesh R , you can have a look in this video , I explained your scenario in detail -- ruclips.net/video/yKzdum_vX7k/видео.html
Hope this will be helpful! Happy Learning
Can we connect?
Hello Namrata Pathak, please post your doubt here , If I know that topic or have any idea , I will surely reply asap , also for any kafka related doubts , please check this playlist once , might help you if the topic is covered in the playlist already -- ruclips.net/p/PLjfRmoYoxpNrs0VmIq6mOTqXP52RfZdRf
@@KnowledgeAmplifier1 hi i am facing one error with my assignment where to post the picture??
@@KnowledgeAmplifier1 what are the rules of data ingestion i.e Kafka to hdfs in which video it is present??