What about outbox/inbox pattern? We may prevent sync communication with the queue and decrease chance of error on receiver side. As service B might want to run business logic on the message first which increases a number of points of failure. Anyway this more a complimentary approach plus to the described in the video.
Thanks for the suggestion! I created a new Playlist on my channel that will automatically be updated to include all of my Software Architecture Monday videos.
Great Video. I'm dealing with another use case for message loss. What if Service B upon sending the ack to Kafka, terminates due to any factor, exception, app pool reset, etc. Or, Service B takes a long time to process the message, Kafka can't wait for any consumer to come back at anytime it wants and only then changes the offset. So what I did was to use a second persistent layer by using Hangfire integrated in Service B. but I hate this solution cause not only it's one more component to worry about, but also it defeats the whole value of messaging since I'm practically duplicating the queue. I don't know if I make sense or not!
Although duplicitous, your technique is a good one if the messaging system isn't available or reliable and you need to store the message and move on, and send it at a later time. You would need some sort of message callback to let the service know the message was processed so it can be removed from the backup queue.
Great videos. Simple and straight to the point
Mark, great video. You answered a question i have had in my mind for a while
What about outbox/inbox pattern? We may prevent sync communication with the queue and decrease chance of error on receiver side. As service B might want to run business logic on the message first which increases a number of points of failure. Anyway this more a complimentary approach plus to the described in the video.
can you please create a youtube playlist for these lesson? this will make it easy for folks to share your lesson around
Thanks for the suggestion! I created a new Playlist on my channel that will automatically be updated to include all of my Software Architecture Monday videos.
I think you missed the DLQ (Dead Letter Queue) pattern here for handling the case that for whatever reason, service B can't process the message
Great Video. I'm dealing with another use case for message loss. What if Service B upon sending the ack to Kafka, terminates due to any factor, exception, app pool reset, etc.
Or, Service B takes a long time to process the message, Kafka can't wait for any consumer to come back at anytime it wants and only then changes the offset. So what I did was to use a second persistent layer by using Hangfire integrated in Service B. but I hate this solution cause not only it's one more component to worry about, but also it defeats the whole value of messaging since I'm practically duplicating the queue. I don't know if I make sense or not!
Although duplicitous, your technique is a good one if the messaging system isn't available or reliable and you need to store the message and move on, and send it at a later time. You would need some sort of message callback to let the service know the message was processed so it can be removed from the backup queue.