Handle Duplicate Messages With Idempotent Consumers | Idempotency
HTML-код
- Опубликовано: 4 авг 2024
- ☄️ Master the Modular Monolith Architecture: bit.ly/3SXlzSt
📌 Accelerate your Clean Architecture skills: bit.ly/3PupkOJ
🚀 Support me on Patreon to access the source code: / milanjovanovic
In this video, I will show you how to implement idempotent event handlers with MediatR. We will use the Decorator pattern to introduce our Idempotent event handlers on top of existing event handlers. The idempotent event handlers will first check if the event has been processed and only proceed if it wasn't.
Idempotent Consumer - Handling Duplicate Messages
www.milanjovanovic.tech/blog/...
Join my weekly .NET newsletter:
www.milanjovanovic.tech
Subscribe for more:
ruclips.net/user/MilanJovano...
Chapters
0:00 What is Idempotency?
0:47 Multiple domain event handlers
2:31 The solution: Outbox message consumer
3:17 Implementing IdempotentDomainEventHandler
6:48 Configuring the Decorator pattern
7:34 How it works 
Наука
Want to master Clean Architecture? Go here: bit.ly/3PupkOJ
Want to unlock Modular Monoliths? Go here: bit.ly/3SXlzSt
Funny. Until today I've never heard of Idempotency and coincidentally I clicked on a video earlier explaining this and now Milan have made a video about it. - Very nice explination, Milan.
Thank you, glad you liked it! Was my explanation easy to follow?
Very interesting, you just opened my mind, thank you very much for the content Milan🤩
Awesome, glad I could be of help Fernando!
With your awesome contents and knowledge sharing. I am feeling awesome 🤩
Thank you very much! I'm glad you like the content 😁
+1 for explaining a good method of enforcing idempotency
Thank you. Did you work with something similar?
@@MilanJovanovicTech yes and though i implemented it differently it was the same principle.
Awesome content. You earned a Pateron
Happy to have you, champ! 🏆🏆🏆
Great content! Thank Milan.
You're almost done with all the older videos 😁
Perfect job for the decorator pattern ;-) interesting video - thx!
Thank you. Did you work with Decorator before?
@@MilanJovanovicTech many times - it often fits when adding cross cutting concerns like logging or caching
A great approach using the decorator pattern, I used pipeline behavior to achieve the same, with different base event class. I have mixed idempotent and non-idempotent calls. But overall, great video.
How did you run a pipeline behavior with events (INotification)?
@@MilanJovanovicTech I didn't. It was for IRequest commands only. I didn't notice that this is for notifications only. You have point. We can't use my approach for MediatR notifications, but I learned something after all. Thank you for your replay.
Nice Video keep it up bro.
Thanks a lot!
If you scale your application, still multiple handler could consumer... Second consumer will throw exception because of duplicate key for consumer table but mail was gone
The question is, if you scale your application. Do you want all instances to be consuming your message at all?
It's a typical race-condition situation.
You would have to think about this differently if you want to scale it.
@@MilanJovanovicTech actually if you use update skip lock when you get outbox messages problem will solved
Good video!
Glad you enjoyed it
Excellent content as always Milan. I have a suggestion for another video not too dissimilar to this one. Here you're using an idempotent strategy to prevent processing back-end messages multiple times, but I'd like to see a 'clean architecture' design for tackling idempotency in an API. As an API vendor I might want to allow a consuming client to provide an idempotency key (GUID) via say an HttpHeader (Idempotency-Key) when making non idempotent API calls (i.e. POST). This key would be used to offer consumers a guarantee that a given operation would only mutate state once and once only. My thinking is that you could create an idempotent unit of work that in turn could insert the idempotency key into a table (where the key is constrained to be unique), and by including this in the transaction scope for a command, you'd achieve the desired result?
Definitely a good idea, and numerous ways to implement it. Agree on the Idempotency header for API side. Internally, maybe a pipeline behavior for commands implementing IIdempotentCommand 🤔
Milan, great video content as always. Are you by any chance planning to show the usage of Entity Framework with the Hi-lo algorithm as part of this series?
We'll see, not high up on my priority list
Nice work Milan. I think you can enter a bad state if after acking the event saving to outbox fails, meaning youd retry it. This should be under tran or two pahse commites I guess?
I think that would be overkill honestly. You'll dig yourself into a bigger and bigger hole if you go down that route.
When would committing the ACK actually fail? Apart from the database being down.
Quick question if i want to implement Idempotency but with massTransit i should decorete the IConsumer interface i asume, right?
MassTransit has an Outbox, which will prevent double publish: masstransit.io/documentation/patterns/transactional-outbox
@@MilanJovanovicTech thx for the feedback! appreciate it !
There are something you didn't mention:
- There must be an unique constraint on the OutboxMessageConsumer table.
- The idempotent handler and the decorated handler must be within the same DB transaction.
Otherwise, the idempotent handler won't work properly in case multiple instances consuming the same message concurrently.
Just a note: Each DomainEvent has its unique Id. I reuse it for the OutboxMessage. That I changed now. Then the Id of OutboxMessageConsumer makes sense.
I got a bit confused because there were stuff made to Gathering that wasn't there in earlier video (or did I miss something)
Yes, there were some changes Marina
- Unique constraint isn't necessary, as I made the PK to have 2 columns (Id, Name)
- I can't understand why do they need to be inside the same transaction?
@@MilanJovanovicTech
If they are not in the same DB transaction and there are two consumer instances consuming the same event concurrently, this may happen:
- Instance 1 check the OutboxMessageConsumer table (return false, which means the event is not consumed yet)
- Instance 2 check the OutboxMessageConsumer table (also return false)
- Instance 1 call the decorated handler and commit to DB
- Instance 2 call the decorated handler and commit to DB
- Instance 1 save the OutboxMessageConsumer entry (success)
- Instance 2 save the OutboxMessageConsumer entry (fail, but the changes made by decorated handler are still persisted)
You need to put them into the same transaction so that all changes made by instance 2 are rolled back at the last step.
what can we do in case SaveChangesAsync fail? for example, after success payment?
Chicken or the egg problem... If you think about it hard enough, there's no 100% guaranteed way to prevent that. Best you can do is add as much idempotency as possible. You'll need a way to check with the payment provider if payment X was already sent.
Nice solution but what if two instance gets same data on same time? In this case, won't the race condition occur? I think yes.
In my opinion most safer waye is using distributed lock with RedLock algorithm etc or If your db supports the select lock mechanism(SELECT ....... FOR UPDATE SKIP LOCKED) ) it will be safer to use and data consistency will be ensured.
Yes, you'll get a race condition. But this is rarely going to happen. In the event it does, you need some sort of locking mechanism, as you suggested
What is the workaround to follow when fail to save the OutboxMessageConsumer record after the execution of the DomainEvent?
Retry, send an alert, move to dead-letter queue
Hello, thank you for the videos, is there a GitHub repo?
The code in the video is shared with my Patreons.
But you can find similar examples on my GitHub
What happens saving the notification to db fails after the _decorated.Handle is completed, next time you will perform _decorated.Handle again, how to avoid it (at 6:50)
Indeed, it would run again. But how likely is that to happen?
Nico work you are a very good programmer but can you explain why you use INotificationHandler
It's from the MediatR library, allows you to publish an INotification that can have multiple INotificationHandlers
Good pattern and great explanation, thank you.
One small nitpick:
I believe you should not be passing cancellation token to SaveChangesAsync in the IdempotentDomainEventHandler, or rather passing CancellationToken.None. Once the underlying event handler has finished work (in this case, email sent), from my understanding we are past the point of no cancellation, since the work is handled but we need to save to reach consistent state. This could probably lead to double handling in rare cases.
That's a fair argument, indeed. I pass it mostly by convention. But would it be realistic for the cancellation to even happen given that this is a background job?
@@MilanJovanovicTech Hard to say, since I am not familiar with how quartz manages their cancellation tokens.
Though the way I see it, there are two scenarios:
- CancellationToken will never be cancelled (ex: perhaps CanBeCancelled is always false for background jobs), so there is not much point propogating the token to begin with.
- CancellationToken can be cancelled (ex: perhaps cancellation is done on app SIGTERM or maybe different background job library is used, etc), so then handlers should be implemented with cancellation in mind.
Either way this is a nitpick, since the timing would probably have to be very unfortunate. Though its something that is easy to fix (at least in this case), so if decision is made to have the token - might as well keep point of no cancellation in mind.
What is the nugget package for Decorate method?
Scrutor
Hi, in background job video you already published event and updated outbox message, till this point event has been published once. Now in this video you are publishing event through decorator, though you are checking it has not already been consumed but outbox message consumer entry was not present for event being checked by this time and it causes same event to be published twice?
Outbox is on the producer side. Idempotent consumers are the consumer side. These are two different sides of the same pipe (the queue).
@@MilanJovanovicTech but same event is handled twice, is that expected or I am missing something
Hi, is there any settings for publising strategy? When i made 5 dummy handlers and 3 trhows expcetion 4 and 5 not hits. It seems that when error occure next publishing is stopped.
Ok, it is NotificationPublisher settings. By default is publishing loop interrupt when exception occure. It is valid behaviour for production ? I expected o consume all possible handlers if can.
With MediatR 12 you have an option to publish in parallel
Hello, what is the definition of IDomainEventHandler and IDomainEvent? How TDomainEvent gets its Id to set into the outboxMessageConsumer?
IDomainEvent { Guid Id }
IDomainEventHandler : INotificationHandler where TDomainEvent : IDomainEvent
@@MilanJovanovicTech Thanks Milan
Hi, I am getting error
"Could not find any registered services for type 'MediatR.InotificationHandlwr
When using services.Decorate from scrutor.
I am using latest version of scrutor and . net 8
Why not just use local memory cache to track this with some cache invalidation time on items so that these tracked "handles" (which are transiently important) don't have to hit a db to check or save - and they aren't so important after success?
What if you want to retry or replay some messages?
Hi Milan, what is the code for IDomainEventHandler please ? thanks
using MediatR;
public interface IDomainEventHandler : INotification;
@@MilanJovanovicTech more exactly public interface IDomainEventHandler : INotification ?
@@MilanJovanovicTech where come from the property "Id" of TDomainEvent notification please ? I try to improve Pragmatic Clean Architecture I recently bought with this feature
Is there any non payable source code available ? 🥺
You can always check out my GitHub
Nice content, pls do share the code
I share the code with my Patreons! Consider joining this month?
I share both the *before* and the *after* versions of the code, so that you can follow along with me during the video.
Hello, can anyone please clarify, idempotency consumers along with publishing events through background job in another video in this series, are raising the same event twice? How can we avoid that?
Do you mean the consumer is running twice? I think that's a DI issue, can't recall exactly how to fix it though.
@@MilanJovanovicTech hi, no consumer is running once but I am using both background job to publish event and idempotent consumer together, which results in publishing events twice, one from background job and another from idempotent consumer. And further I could not use scruter library to register idempotent handler so I used inbuilt services.Scoped(...) method to register it, if it makes any difference.
@@MilanJovanovicTechNo but I am running a background job which publishes events and idempotent consumer together. So event is processed first when background job fetches outbox message and second time when idempotent consumer handles it. This way, an event is processing two times.
@@MilanJovanovicTech it seems resolving now without additional changes. I just tried again scrutor library which was first day giving an issue, this time same version of library seems working as it should. Though not sure, what happened but it works at the end as it should. Thanks!
I would have add this name and Id in cache like key. After proccessing, remove this Key. The speed is better and don't accumulate database with multiple connections by request.
But the consistency is a concern. And what will happen in a distributed environment?
@@MilanJovanovicTech I think that be variable by scenarios. For example, I add that idea in invoice system to town hall, and work it.
If persistence is really needed, I think Redis solve this problem.
@@MilanJovanovicTech Also, can you have a community in Discord?
TIL that you can cheat at Visual Studio
Interesting that some haven't tried that while debugging