Apache Kafka 101: Consumers (2023)
HTML-код
- Опубликовано: 6 окт 2024
- ► TRY THIS YOURSELF: cnfl.io/kafka-...
In this video you learn about Apache Kafka consumers and consumer rebalancing. Youl also learn how to use the consumer API and the KafkaConsumer class to connect to the cluster to subscribe to one or more topics.
- -
► For a COMPLETE IMMERSIVE HANDS-ON EXPERIENCE, go to cnfl.io/kafka-...
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness the business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics, and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large-volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit confluent.io
#kafka #kafkastreams #streamprocessing #apachekafka #confluent 
Наука
Best Kafka educational videos out there. thank you
Each partition ensures the order of the messages contained therein. The producer can target messages to be produced onto a specific partition by the message key. Consumers can target a specific partition by specifying the key of the messages they're interested in. If the message order doesn't matter as in when transactions are idempotent(i.e. Order has shipped) then you can produce those messages in a round robin fashion to multiple partitions and have each partition assigned a consumer where each consumer belongs to the same consumer group, consume those messages at scale. 10 partitions then define 10 consumers within the same consumer group so that messages are consumed at scale of 10. Got it.
The MOST important thing to know is if msg Order IS important for message consumption(i.e. CRUD event notification). If it is then you can only have 1 partition and have multiple consumer groups each with 1 consumer reading from that partition. If msg order IS NOT important then you can scale.
Very important aspect indeed. Consumer scalability is not elastic when message ordering matters. I think though in some cases one might sacrifice consistency for performance and scalability by using compensation transactions after processing instead at the Kafka consumer groups level in order to account for order, but it’s not always possible.
What is the importance of key schema from consumer perspective? Is key important for the consumer within the application?
Interesting point about applying ordering guarantees by forcing only 1 partition to be present. Thank you!.
I'd add that even if message order is important to keep, you might still be able to scale your consumer group. If ordering doesn't necessarily matter across your entire log, but by some logical grouping of subsets of messages.
I.e. I have a Topic with CRUD events from Customers A and B, and for each customer i want to preserve message order, but between customers i don't really care.
Then I can for example pick "customerId" as my message key (or if necessary, implement some custom partitioning logic), ensuring that connected events where I actually do care about ordering will land on the same partition, thus be consumed in order, and still have 2 partitions with 2 consumer instances.
Continuing the example it would be e.g.:
- Partition 0 holds the events of Customer A - in order, and
- Partition 1 holds the events of Customer B - in order.
And since partitions are distributed between instances of a consumer group, one of my consumer instances will process the events of Customer A - in order, and the other one will process the events of Customer B still in order.
In such a scenario you can have multiple consumer instances in the same group, while still retaining sufficient ordering for the messages.
Wow - a totally perfect video, clearing most of my doubts in less than 7 minutes!
Thank You!
I like the way Tim explains , straight to the brains
Great video! Any tips of:
1) how to define and monitor a good number of partitions?
2) how to handle consumers failures? For example, how to not get stucked because of a miss-formed message?
I guess you already figured everything out as you asked two years ago, but if you need a web console for managing/monitoring topics/brokers/clusters then provectuslabs/kafka-ui is a very simple. open source and easy to install solution. With docker it will take few minutes to install and run.
Tim has explained kafka `simplified` exceptionally well.
Hello, great video. I'm trying to find a strategy to resolve transactions of microservices communicated to each other by Kafka. Do you think that is possible? Thanks for your help.
Great videos!
How do we scale(on what criteria) consumers in consumer group(considering we have 3 partition with 1 consumer to start with )? is it message count in topic and auto scaler decides to add in new consumer service ?
where i can bought that t-shirt! it's so cool!
We run Kafka consumer on Kubernetes. For one use case we use pod-name as the groupId to filter out specific events. Is having 1000s of groupId for a topic an issue? will the Kafka cluster cleanup unused groupid safely in the background (metadata and offsets)?
Very nice
I have a question, because of two instances in the same consuming apply (group) will be assigned to differences of partition right ?
so could i it miss messages from partition it not assigned ?
A consumer in a consumer group is definitely assigned to a particular partition. The ideal scenario is that the number of consumers in a consumer group is less or equal to the number of partitions so that there are no consumers being in idle state.
Even I have same doubt, if multiple instances are consuming, and getting messages from different partitions, won't that miss the messages from the partition from which it is not distributed?
Where can I get hands on api examples
Hi I have a doubt in this partition
Number of consumers is restricted to number of partitions?
Number of consumers in a consumer group is limited to number of partition. You can have multiple consumer group consuming from one partition
Can I have the consumer act as a producer(publisher) as well? I have a user case where a consumer (C1) polls a topic and pulls messages. after processing the message and performing a commit, it needs to notify another process to carry on remaining work. Given this use case is it a valid design for Consumer (C1) to publish a message to a different topic? i.e. C1 is also acting as a producer
use kafka streams
Complicated!!!