Topic vs. Topic: Solace PubSub+ and Apache Kafka
HTML-код
- Опубликовано: 15 май 2019
- Solace Developer Advocate Aaron Lee explains the differences of how topics are implemented in the Solace event broker and the Kafka streaming platform. This is not a comparison of the products as-a-whole... just simply how each of them have implemented the publish-subscribe communication pattern very differently, including the concept of "topics".
Learn more about Solace vs. Kafka:
🔗Solace Topics vs. Kafka Topics: What’s the Difference?: tinyurl.com/y2b9ajaa
🔗Solace PubSub+ vs Kafka: The Basics: tinyurl.com/ydz87hh2
🔗Solace PubSub+ vs Kafka: Filtering: tinyurl.com/a9px2buu
🔗Solace PubSub+ vs Kafka: Multi-Site Architecture: tinyurl.com/5n7uehwe
Why you should look beyond Kafka for operational use cases:
🔗Part 1: Filtering and In-Order Delivery: tinyurl.com/bddfpf7c
🔗Part 2: Flexible Event Filtering: tinyurl.com/bdd2pykj
🔗Part 3: Securing the Eventing Platform: tinyurl.com/ysewx2x6
🔗Part 4: Streaming with Dynamic Event Routing: tinyurl.com/2p8cb39d
Music by Frakum, licensed under Creative Commons: freesound.org/people/frankum/... 
Наука
Fantastic! This clears up things quite a bit especially from the perspective of typical usecases where Solace would be a perfect fit 👍
Informative and short. Thanks
Excellent explanation, thanks
thanks! is was really helpful!
Nice explaination... very nice
thanks!!
Excellent explanation and nice surprise to see Ottawa's map used in your example!
Ottawa represent!
do you have tutorial on how to do Auth0 in Solace using C API?
Great video. You just walked me off the (Kafka) ledge for the app my team is designing. We really need the dynamic topic creation part (Forex example in your video). I do have two follow up questions
1) For my app design my topic can be [app] / [userid] / [sessionid] / [reportid]. I have 1000's of users logged in on any given day and multiple sessions (may logoff and log back in) per user. Given the number of Topics that can get created, (a) do you see a performance problem (b) do I need to delete the topics when all messages in a topic are processed
2) A unique need for my app dictates that I need to know if each topic has the certain number of messages that I expect. reason being, each topic consists of messages (say message header has 1/5 or 3/5) that have been chunked for parallel processing. So at then end I need to know if i have received ALL 5 messages before I can start the task to reconstitute the final output
Hi @Shanki L - I'll take a stab at your questions!
1. Nothing to worry about - in Solace topics are just metadata on the message so you can essentially have an infinite amount of topics over time. Also since topics are just metadata and not an actual object configured on the broker there is no need to delete them.
2. We do have a mgmt API (SEMP) that allows you to see the current queue depth, but in general for a use case like this you would want to implement some sort of business logic in your app as every situation is different. For example, are you guaranteed to only get message 1/5, then 2/5, then 3/5 or could your app be receiving other messages as well? It is likely that this use case may require the ability to filter/partition events across multiple consumers that can still receive the events in order. This blog defines how you can do that with solace: solace.com/blog/sticky-load-balancing-in-solace-pubsub-event-broker/
Sorry for the delayed response! It's easy to sometimes miss questions on youtube videos. If you have more questions feel free to join us over at solace.community as a lot more people keep their eyes on the questions that get asked over there.
Great explanation. Thanks. Is there a chance to see short HowTo videos that highlight the most powerful parts of solace? It would help developers to easily get into the product and facilitate its adaptation to real business scenarios
Hi Vitaly... yup, we're gearing up to start rolling out more video content to help devs easily get started on Solace. Thanks for the comment! -Aaron
Kafka ksqlDB does give you the same functionality as the hierarchical topics solace has. I would want to see the performance of simple topics vs /hierarchal wildcarded topics at scale. I have a feeling for performance reasons these would be discouraged, and would fall back to a partitioning strategy. Are there any performance metrics like for like Kafka vs Solace?
Hi Matt... sorry for the delay in replying! Yes, there is definitely similar functionality that can be achieved using KSQL, but how it's implemented is very different. KSQL requires all messages pulled out of a topic and compared in client/application space (outside of the broker) and then re-written to a new filtered topic. Essentially doing content routing/filtering. But having to pay the network bandwidth of messages going from the brokers to the KSQL processes (possibly multiple times for multiple queries) and the CPU overhead and finally the extra disk space for the filtered topic. Solace's topic filtering is all done inside the broker, and is highly optimized for the basic wildcard/regex pattern that's being matched on. Further to topic matching, Solace also supports Selectors, which examine the message's User Property Map for even more detailed filtering requirements if necessary. So essentially, the Solace broker can provide two levels of filtering before the messages have to transit into the client space for more advanced content filtering like KSQL.
And in terms of performance, there can be some impact to using a massive hierarchy and poorly-chosen structure (e.g. with regards to how wide the hierarchy is at deeper levels), but typically we don't see that "in the wild". But still, the performance of doing pattern matching on a string max 250 chars long (the Solace topic) is obviously going to be better than doing content filtering on every message's payload contents which need to be deserialized like in KSQL.
Happy to continue this conversation further over on solace.community/ come find me there!
In Solace, there are two options (exclusive and non-exclusive) for topics creation. I think, you explained only exclusive way where topic subscriptions are at event brokers? Non-exclusive way is similar to Kafka ?
Hi there @pahal1858! I want to point out a couple of things before answering your question:
1. There are no topic created with Solace! Topics in Solace world are a property of the message set by the publisher. It's important to get this concept cleared before dealing with Solace topics. Check out 3:40 in this video
2. You can create queues on the Solace broker; this is where the exclusive and non-exclusive concept comes in. Read more about solace queues here docs.solace.com/Messaging/Guaranteed-Msg/Queues.htm
Now to answer your question, Aaron did not cover queues in this video! I recommend checking out this playlist if you want more input on queues ruclips.net/p/PLY1Ks8JEfJR57vCkrQK0Y9bn8DKNMMJI_
In a nutshell, queue exclusivity is a property of the queue and it boils down to how many consumers can bind to the queues in an active state (note the keywords used here: "bind" to queue instead of "subscribe "or "connect" to queue).
- Exclusive queue --> only one consumer can be in an active state when bound to the queue. Message order is preserved
- Non-exclusive queue --> multiple consumers could bind to the queue in an active state. Messages are delivered to all consumers in a round-robin approach
Note that queues on the solace broker can also subscribe to topics! In this case, the publisher only has to publish messages to pre-defined topics. Hope this helps!
-Tamimi
We’ll done!
What happens when the consumer is down and the event arrives at the broker in case of Solace?
Hi Ashish. This depends on whether you have published the event (Message) as Direct or Persistent. If you need the message to be Guaranteed delivery to the consumer, then: A) the publisher will set the DeliveryMode to Persistent, and publish the message to a topic; B) inside the broker, configure a Queue for persistence, and add a Topic Subscription to that Queue that matches the topic of the message; C) the consumer will connect to the Queue to receive its messages. This pattern ensures messages will always be available for a consumer, even if it is offline. See these links for more info: solace.com/blog/topic-subscription-queues/ and docs.solace.com/PubSub-Basics/Publishing-Guaranteed-Messages.htm and solace.community
kafka client api's also have call backs build in.....
Good to know! Solace also has blocking/synchronous publish and receive calls as well. But this video was more about how the brokers implement topics, the API diffs were really not meant to be exhaustive.
EDIT: I took a look, and can only find callbacks for things like rebalances, security, errors, and async commit notifications... not for message/record receipt. It appears you still can only poll() for messages.
Finally i saw real jimmy Neutron 👻