You DON'T want an In-Memory Event Bus like MediatR
HTML-код
- Опубликовано: 22 май 2024
- An in-memory event bus is not a good idea. Typically you'd see an in-memory event bus used for publishing domain events to notify other parts of your domain, which is done in-process. If you're in the .NET Space, an example of this is the library MediatR. Let me explain the pros and cons of an in-memory bus and why the downsides far outweigh the benefits.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via RUclips
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:41 In-Memory Event Bus
3:40 Example
5:21 Retries
7:31 Out of Process
#softwarearchitecture #softwaredesign #eventdrivenarchitecture 
Наука
I think the term "in memory" is a little bit misleading because this is not the problem here. The problem is the synchronous processing. You can also process everything asynchronously from in memory, the only problem is, that you loose the message on an app restart and this is where Hangfire for example comes in.
Exactly. Another benefit of an in-memory queue is that you could swap out an in-memory queue with an out-of-process queue if you ever want to split up a monolith, without necessarily changing the interface. If you are careful with the queue interface, you can do it without code changes to the producers/consumers.
I'm using MassTransit and the Publish() method is asynchronously so the application service will not wait for the publish method to be finished, he just call it and go next
The video is a bit misleading. It's not Mediatr that is the problem here but the fact that you were using Domain events in places where you want to use Integration events which will trigger listeners that are out of process. You'd use domain events when you want to modify another aggregate from the same bounded context as a result of a domain event ..and that all is part of a single db transaction. IMO for that matter Mediatr does the job just fine.
Not intended as misleading. It maybe that I don't view domain events or publish/consuming them that way even in-memory as a good solution.
@@CodeOpinion Np, I understand we all have different opinions. I used the term "misleading" as the title and the overall message is "don't use it", while it really depends on what we are doing and how we do it. We could also handle domain events asynchronously using messaging and the outbox pattern but sometimes the extra effort is not worth it. Although I have a different opinion on this one, thanks a lot for putting out great content, always a pleasure to watch it 👍
@@CodeOpinion I have to agree that the video is misleading. You make a clear generic statement in the video title and try to justify it with a wrong example for which MediatR should not be used. This is quite misleading. The question is do you need immediate consitency or not and this is a business decision. Just take the financial market where you find many cases of required immediate consistency even spanning multiple systems. If you have a limited number of products that you can't reproduce (e.g. tickets for an event) making the stock eventual consitent is disaster. You have valid points. They would be more valid if you use them for the appropriate cases. For people watching your video it would be more helpful to show the various options and requiements and show the possible solutions for those cases.
@@CodeOpinion I really wish you would rename this one. The naming is bad. I agree it's misleading and it really had nothing to do with MediatR
I agree and disagree.
There are two types of events as far as I am concerned.
There are domain events which are handled within the domain and need to be part of the business transaction.
Then there are integration events which are outside of the business process.
Or, you could call them internal and external events.
An example of an internal event in our app would be when an Invoice is created, the PO needs to be updated. Those are two different aggregate roots. So, an InvoiceCreated domain event is handled by a handler in the PO feature that updates the po domain object and it gets saved as part of the same db transaction. This is a business process that should succeed or fail. If we want to publish an external event we use the outbox pattern, and there is a handler that put the event into the outbox. The outbox processor will raise the event as an integration event.
Thoughts?
The purpose of being in-memory is to get immediate consistency, eg using the same underlying transaction as the producer. If that's what you think you need, then sure. However, I won't convince everyone, but you can embrace the asynchrony the same way you would if you needed to perform a long running process with multiple different logical boundaries involved.
Agreed, I usually use an in-memory event bus for things that I can save as a part of the same transaction. It's much easier than external event bus which may require things like outbox pattern, handling duplicate events, retries, rollbacks, etc. You can also avoid having an external event bus at all if your app is a monolith and doesn't have integration events or async workflows.
@@CodeOpinion I think my point was, it's not a black-and-white don't do this, do this instead type of situation.
Totally agree with you. Let’s be serious, business should be consistent in the same bounded context. I didn’t meet any business owner who said to me: it s ok to make an order and fail updating stock, all want consistency at least in the same domain.
So, in my point of view, in memory event bus is good, it’s nothing bad at it. Yes, you shouldn’t send email from the same unit of work/tx, but persisting aggregates and keeping them consistent is a must.
Life beats movie! I’m tired of hearing all these fancy stuff about async and in the end having a lot of problems in such a system in production.
But why though?
For your "internal events", just have an interface with a descriptive method - use your DI container to register a list of instances of that interface. It's much simpler and clearer, and you don't need any event facility - your method calls are "events", your registered instances are "listeners", whatever calls those instances is the event "publisher".
It's just composition. It gives you all the same decoupling, flexibility, separation of concerns - without any framework or facilities, just your domain.
Unless you have a long term goal of moving to microservices, I wouldn't bother with this - and if you do have that goal, as covered in the video, you should probably just go for a proper message queue right away.
Jimmy Bogard's "6 little lines of fail" is a more in depth talk about these problems and worth a watch. I think the in memory event bus is a red herring here, inline those event handlers and you have the same consistency problem, conversely you could solve the problem with in memory synchronous handlers too.
I think event driven seems harder because these consistency problems are acknowledged, there's a pervasive and naive belief that you can just slap things in db transaction and call it a day but as your demonstrating that's absolutely not the case.
What you (rightfully) call "in memory event bus", like mediatr or spring boot ApplicationEvent, are just implementations of the Observer Pattern. When you are in a monolith you do want to do stuff in the same transactional boundaries: that's why you've chosen to be in a monolith in the first place (or at least you should had to) what you don't want it's coupling and these "local" or "domain events fit pretty good in the use case because you can trigger many pieces of diverse workflow without the caller knows it. No eventual consistency, no scalability but no stress to think about designing a distributed system. When your requirements change and you want to separate transactional boundaries, local/domain events can be easily turned into remote/integration event. It's important to stress (like you did) that this transition can only happen by integrating a process that it's able to durably store events
Just started using mediar and now you come with this warning. but now that i have watched it, i am safe. I use mediator to publish progress events from the appservices to the frontend to update a progressbar. But it is good to know that i should not get carried away with the possibilities.
I had a very similar situation with sending emails from one of the MediatR's handlers on one of my previous projects. But what I think is important here is to draw the right boundaries between the components. Something that's not essential for the operation to complete doesn't need to be in the same transaction and should occur asynchronously.
Agreed. The issue is knowing when you're writing an event handler, how it's being executed. Is it in-line or is it async? If you have a very clear distinction in code, then I could get behind that. But if you don't, then you have no idea when writing a consumer how it's executing and how you should be handling failures.
Would you recommend MediatR for replacing in-App service layers?
We have had several problems with service layers, like circular dependencies, tight coupled behaviors and so on and we´ve found it easy to replace all of it with MediatR.
In general, we replaced something like a "ProductService" with a bunch of commands and queries and ended up putting the business logic right into them. I mean, like always, it depends - but in our situation, we´ve found it quiet special.
I think you can also replace Repositories with the same strategy as Queries becomes very strict in terms of "specification".
We did it and have not regretted it for a second. It made our systems much easier to maintain and test.
I still like mediatr for batch scripts and general application flow.
What do you think?
Saving the events to a database and having a background job processing them would be my preference. Then it can retry without blocking anything if it needs to.
In your example using a durable storage, do you execute the main logic and in the same exactly process store in some table or anything like that atomically? And then use a job scheduler from HangFire to perform the logic of sending email? Or use another feature of hangfire?
This part was not clear for me watching the video. Could you enlighten me?
In Java/Spring you can annotate a 'collaborator' to control transactionality. Like only outside of a tx, starts a new one, only part of an existing, etc. Can MediatR not do that? Seems like you could do something like the outbox pattern if you're in a txn anyway. It feels like there's ways to suppress blowing up the txn if that's really an issue.
I don't like how it makes simple questions like 'where is this referenced? what's the flow? are these fail fast?' take an extra level of indirection.
It might be a good way to start journey with inversion of control and establishing how subparts of the system interact with each other - "robustifying" handlers might help (handling errors gracefully, retrying), but doesn't make "problems" disappearing.
As always, thanks for sharing your opinions, much appreciated!
@CodeOpinion I do like the idea in general, but what would happen, let's say IF you in an event change the status to '1' and fails (user did not notice), user changed the status to '3' and that succeeed, because, well it's a better status for our task. The failure will remain in the queue right, wouldn't this change that status to '1' after it recovers? that's not something that we want? how is this handled?
Why not just use seperate transaction for individual handler?
Would you recommend this approach for an application that has a workflow (e.g., one that has several steps the user has to accomplish to finish a task) or is there a better approach for something like that?
Orchestration or Choreography ruclips.net/video/rO9BXsl4AMQ/видео.html
What would you advise to use on AWS for out of process event handling, both in a monolith and outside of it?
One approach would be SNS + SQS, but then each handler would need its own queue to make the processing independent and retry-able. Thanks in advance!
Use a messaging lib like nservicebus, mass transit or any other that sit on top of sqs and sns.
@@CodeOpinion Could you make a video about it? :)
Appreciate the video as always! Although, I'm not entirely in line with you on this one. I think comparing an in-memory event chain using a shared transaction to an orchestrated one as mentioned here does not really do justice to the added complexity of a distributed transaction / saga orchestration necessary to achieve the same logical transaction on an event queue level.
I'm definitely with you on favoring an out of process event bus, but I think something like MediatR has a valid use case as in serving as an observer implementation to decouple code sections instead of introducing a complex chain of direct dependencies between modules. I fully agree though that you definitely need to know what you are doing, and when to maybe implement handlers in a fire and forget fashion.
Bottom line, I like to use it and think it absolutely deserves its popularity, but as you mentioned, I think it is not necessarily a great choice when dealing with critical domain events due to the fragility you described very well.
This is more of an it depends for me. Either of the consumers should really be out of process given the nature of their job, but the behavior you’re describing could be the desired behavior for other processes if you’re staying internal to that boundary where a domain event might be more appropriate.
What I see to be the problem is actually failing before the transaction is over. Ideally, if one really wants an in-memory event bus that executes it's handlers in isolation, the producer has to be in a separate process/IoC lifetime scope to the components consuming that event.
With MediatR specifically that's achievable by implementing a custom Mediator overriding the behavior of publishing to enqueue it in a different thread for example
But that only solves the transaction issue... With regards not losing events then only a persistent event bus/out-box pattern would solve it
What "out of process" message queue lib is recommended for migrating from Mediatr? Hangfire?
Task Queue like Hangfire works. For messaging, NServiceBus, MassTransit, Wolverine, Brighter, Rebus, CAP, they can all sit on top of different transports.
It seems to me like the mistake here is in the design of the error handling, which you kind of mention at the end of the video, but still identifying process boundaries with the scope/reach of the exceptions. I think that's a confusion. In the example, the error should only bubble up to error handling logic that exists at the granularity of the event, not that of the process. Then, the error gets logged or whatever you have to do, and the publisher can (and already has) carried on with its work without worrying about subscribers, as it should.
Orchard has integrated an in-memory event bus for the purposes of decoupling, inversion of control and extensibility from its first version, and it's been working great.
I think a great strategy is to implement the in-memory hub using the exact same abstractions that you would for an out of process one. Even better, make the transport pluggable, so that in-memory is just one option of transport, and changing it can be used as a way to scale out (or in) the application without having to redesign it.
I've used that approach in internal products, and it has the advantage that you can leverage it to start decoupling a monolith progressively without changing its process model at first, introducing events little by little in-memory. When time comes, you can replace the transport and unlock new scenarios.
Hey Betrand, thanks for the comment. I was away, sorry for the late reply. The gist of the video, maybe not explicit enough (my bad) is that when events become critical, especially to workflow of a long running business process, you can't just lose the event because of a failure at the consumer. Meaning, if a consumer fails, you can't just disregard and log it. Logging the failure does nothing. You need the event in a form in durable storage where you could easily modify and/or re-process it. Nor should be you having to write specific error handling logic for every consumer. If you move the message to durable storage and out of purely in-memory, you never lose the message. If you restart the process, you lose everything. Durable storage, no problem. Messaging libraries like NServiceBus, MassTransit, Brighter, Wolverine provide all this functionality out of the box for resiliency (retries, backoff, outbox, DLQ) as it's common place for messaging. And yes, agreed, most of them have in-memory transports, but it's more for local dev/testing.
@@CodeOpinion yeah, thanks for the answer, but my point is that you don’t have to lose the message, memory bound or not.
I use it in a desktop app to streamline clicks and complex actions.
What about having each handler run in parallel and any errors are swallowed and logged when not handled? I'm having that solution. This mitigates the coupling and still is in process.
Assuming logging that is good enough. When events are critical to the system, you'll likely need to re-process them if they do fail. You'd need a way to achieve that from your logs. Storing the messages somewhere durable solves this. Eg, moving out of process/memory.
Then when it fails you need a developer to go find it in logs and rerun the values instead of having those failed events in a queue, like Derek says
In general, if some business process requires atomicity I'm ok with a shared transaction when publishing in-memory, but I agree it should not be the default choice.
IMO, the main issue in this demo is its architecture hiding the underlying transaction instead of making it explicit that a transaction is being used. After fixing that, some handler making a "synchronous" call to another service within a transaction should be obvious (and clearly a problem for someone experienced).
This has nothing to do with "in-memory" or not. Of course, you have the possibility to share transactions only becasue you ar in-process. But you still don't have to. Just becasue you message bus is in-memory, you still can decouple them as you like - and should. Event though you are not forced to.
In some of your past videos you have used MediatR. Has your opinion changed since creating those videos or have you always had this level of skepticism? If your opinion changed, what did you experience recently that made you change your mind?
Nope, I still it for commands and queries. Just not for events/notifications.
@@CodeOpinion Ok that makes sense. Thanks for the awesome content!
Thanks for the video. We're using MediatR as an In-Memory EventBus just like you describe. What I like about it (as you say) is that I can have the whole operation within a single Db transaction. If something fails, e.g. in an event consumer, we can roll back the transaction and prevent having the database in an inconsistent state. I understand your point - you don't want a single failing consumer to block the entire operation. I guess you could use the Outbox Pattern as an alternative to a single db transaction. However, this will add its own complexity. Depending on how you set it up, and using something like CAP, I believe that the Outbox message is 'guaranteed' to be consumed 'at least once'. Therefore, you'll have to add additional checks to ensure the message isn't consumed more than once. Thoughts?
Yup. I have a bunch of videos on this. Most recently: ruclips.net/video/anL-RGKz_Ak/видео.html
Of course, consumers should be idempotent. Either by design or by nature. I would make a case that they should be regardless whether you use Outbox pattern or not.
I think it depends. We also have MeditR as an In-Memory EventBus and we want to rollback whenever something didn´t worked well. In your case - and that´s a fairly good reason, you still want to get the money and that´s why you should think different. However, my case requires to let the user know that there´s something wrong.
The only In-memory event bus I see as acceptable is for testing purposes and you're trying to avoid mocking the entire bus
Really?
Hi Derek, I really like your videos and globally your channel. Thanks a lot !
Do you think provide your own English subtitles ? These from RUclips are not the best quality.
I do not provide my own at this time. Hopefully in the future I'll be able to outsource it and have them done.
Thanks
The only use I can see for In-Memory would be for events that don't really matter - things that are not critical to your application if they don't arrive. For those, there might be a lot of performance gains to be had.But it would be rare.
Like some sort of custom monitoring or analytics kind of thing
I feel logging is such a use case
@@parlor3115 though for logging you'd probably be better off logging to console and having another process scrapping those logs and shipping them do whatever centralized log sink you use
I agree with you, except it is not rare. There are lots of use cases that require message loss tolerance.
wouldn't running all message/event consumers in their own thread solve that issue? it would still be all in memory, same process but different threads, therefore, as i understand it, separated...
Yes, but it's not durable since it's in memory. Any process restart you'll lose any events you needed to process. Also when it comes to failures consuming events, if you generally want to them stored somewhere durable so you can manually take action against failures. (eg dead letter queue).
Well, you are putting all of it into one context.
You cannot compare domain and integration events.
Domain events should be done all or nothing and integration events should be processed via outbox, so they roll out only when all the domain events succeed.
The subject of the video is very unfortunate. It's not about where the bus exists, but rather about the (a)synchronicity of the listeners.
If you have a threadsafe queue and listen to the messages in separate threads, you'll be fine. There are use cases for this as opposed to introducing an external persistent message bus.
For example, an MVP or a prototype for a new system.
You could go even further and say that with proper error handling around listener invocations, it's sometimes fine to run in the same thread.
If the event is only in memory, even if you fail in another thread, you don't have that message anywhere durable. The process restarts, you've lost the event. Moving the event out of process allows you to execute in isolation and fault tolerance.
@@CodeOpinion That's true 👍
in my opinion thats not a problem of in-memory or not, its more of a problem of the mediator-library and how it is dealing with forwarding events to consumers. Even in-memory or in-process the lib could call the consumers in an async way, and also wrap each call in try-catch, so other consumer call's won't be bothered. Of course if in a lib like mediatr, there is a very trivial synchronous implementation of the publish logic (like for-eaching over a list of actions) then this will cause your described problems.
It has everything to do with being in-memory and not being durable. What's the recourse if an event handler fails? Retry it forever? Just log it and hope for the best? If events are critical to your system, you need that event to be durable so you don't lose it (eg, DLQ). Restart your process? Event is gone.
I agree completely! I would also like to mention that as more handlers are being added, less performant your transaction will be. The bigger the tranbsaction, the more likely it is to face locking problems, gap locks or even deadlocks. Also, a persistent out of bounds memory bus gives you resiliency against failure. It decouple your operations in time and space
There is one place where i would use an inmemory bus and it is to sending notifications via websockets or something to the UI. Something that is fast and, if it fails, nobody really cares
Reminds me of Jimmy Bogard's talk: ruclips.net/video/VvUdvte1V3s/видео.html
Haven't watched it, thanks for the sahre.
You DON'T want an In-Memory Event Bus like MediatR... Yes you do. Not all events need to be durable. Maybe you're just feeding some frames to a streaming service. Who cares if you drop a few frames? Maybe you're feeding IOT data, like temperature sensor readings, to a device, like a thermostat. Who cares if you miss a few readings? This is also why UDP exists. Not every message has to be delivered. Not every event has to be processed. Even external queues, like RabbitMQ, have variable durability options. Performance vs durability is just a tradeoff you make based on your use case.
Outbox pattern
Ultimately, durable storage of events.
The described problem arises not from using an In-Memory event bus, but on the implementation of the handlers.
could you elaborate on how would you implement the handlers from the examples in the video differently?
I disagree... (Sorry for my english) I would argue that your problem here is not the in-memory event bus but badly defined transactional boundaries.
You start in Order domain and publish a "DomainEvent" from Order Bounded Context. Domain events are internal to their BC and can be published in-memory and maybe even usually be processed syncronously (if part of the same transactional boundary; sending email etc should be executed asyncrounously outisde the aggregate transaction). The other two operations (run by your event handlers) belong to other bounded context(s) and so shouldn't handle an internal event from other BC directly. Domain events shouldn't be part of the public contract. Your two "event handlers" shoudln't handle the DomainEvent but an IntegrationEvent (external) published in reaction to the internal event. Those can be published in- or out-of-process and processed asyncrounously. Integration events are part of the public contract. In the video you treat all your events as internal even though they are published outside their BC. The problems described in the video aren't a problem when you apply the distinction between internal and external events and process them accordingly. This way the order will be created and the other two operations will be eventually executed if creation succesful and external event published.
Honestly if you are writing enterprise code this complicated please rethink your architecture. I used to write code like this 10 years ago, but since The Cloud no one deployable unit ever gets complicated enough to need complex libraries to manage all the other complexity.
In-memory does not imply in-process.
Starting with this assumption it all went downhill from there.
I'm memory, implying not durable.
@CodeOpinion everything is in memory at some point. It's only a matter of ordering of operations: first commit to disk, then reply to the caller "got it".
You can have consumers in different threads, in-memory, syncing to disk along the way, and handling their own crashes. And yes it doesn't have the drawbacks you've mentioned.
Also, that bus is still abstracted away, and the volatile implementation makes for a great test double.
All in all, all this debate for nothing, for me it's just an implementation detail.
I'm not convinced, to be honest. First of all, opening db transactions in the business layer seems smelly already, ideally the business layer shouldn't even know about databases. Second, it seems that your problem is more isolation than messages. Your consumers should not affect each others and neither the producer, true, but that can be done by handling exceptions in the dispatch and enforcing some guarantees in the consumers. Besides everything else, randomly rolling back a transaction because a consumer threw an exception seems questionably, ideally you roll back in very specific cases and not as a catch-all solution.
Handling failures in a consumer is the challenge, which is why it being in memory is the problem. If a message is failing, just log it? Not really if you're levering events as workflows. Unless you want to roll your own storage and then re-dispatch of an event... or just use a messaging library on top of a broker for durable storage or a task queue.
@@CodeOpinion Yeah, isn't message persistence yet another orthogonal issue here, that is perfectly doable where necessary even with an in-memory hub?
One benefit is that the consumers can choose to be part of a transaction or not.
They don't get to choose if they don't know there is one wrapped around them.
@@CodeOpinion but you have the option to let them know.
Excellent. I really don't like Mediatr. It adds little value especially where services are 'hardly' coupled (as I liked to call it), that is where services can still be aware of their callers and not agnostic of them. If Mediatr was just sending a message to a handler that simpler pushes onto a service bus then fine, but then why bother having Mediatr, I can just publish directly.
I strongly disagree with this. Nothing pisses off a customer more than a system that 'sort of' works.
Sort of works would be an event failing to be successful consumed. That's exactly what the problem is when you fail to consume and don't the event in durable storage that you can retry or put to dlq for manual review.
@@CodeOpinion ok, put the original message in durable storage and then consume it with mediator or whatever pattern. Just move the pattern in question to consume what's coming out of your queue.
The entire argument is a straw man: if you do stupid things in a way that you can only do with in memory (really in process) buses it's not the in memory aspect that's the problem...
Misleading video!!
What do you think is misleading?