I agree with you, when I spent a lot of time targeting functions with isolated model ( no webjob dependencies) I don’t want to bring in webjobs to get the durables orchestrator to offload the work.
I would do an RPC call to the orchestrator, let it orchestrate everything, and then reply back. The downside is that the producer will be waiting. This approach is fine in a service as part of an asynchronous process in a service. Like within a consumer or separate process. You should not do this call from a Web API in order to get a result back since it might take ages for responding back to the client. That defies the purpose. In that case you would perform the RPC call and when you get the reply you will send it to the client using WebSockets.
I'm new to this topic, isn't the same problem present at both 2:06 and 4:38? I.e. in both situations, whether using synchronous or asynchronous request/response, your system will be left in an inconsistent state if the warehouse fails to respond? Is the benefit of asynchronous (e.g. using an orchestrator + message broker) purely just retry-logic? Also what happens if the message broker goes down? Aren't you creating a critical dependency between all systems? Also what is the lifecycle of the orchestrator? Is it an object that gets created for each new request and garbage collected when it receives all the responses it's expecting? What happens if one of your systems goes down and it starts creating heaps of orchestrators that fill up memory because they're not getting responses? Sorry for the dumb questions, just trying to figure this all out.
Great set of questions. While it seems like they are similar, when you're developing it to be asynchronous, there isn't any expectation of immediate consistency. In other words you know while the workflow is executing that time isn't apart of the mix. As your example if the warehouse isn't available, that doesn't mean the entire workflow fails. However if it were synchronous, if the warehouse weren't available the workflow/process does fail. Yes the retry logic is part of it but it's also removing that both need to be available at the same time. You can place an order and both billing and warehouse don't need to be available. Yes the broker becomes important, but for reliability in not losing messages you'd use the outbox pattern: ruclips.net/video/u8fOnxAxKHk/видео.html The orchestrator can contain state and depending on the library you're using will depend if it's in memory or getting created for each invocation. The alternative is event choreography, check out this video: ruclips.net/video/rO9BXsl4AMQ/видео.html
@@CodeOpinion okay so is the purpose of an orchestrator (over event choreography) purely to increase code readability and the ability to reason about your code base? The implication being that it becomes really hard to mentally understand a business process by jumping through "hoops" of disparate events littered throughout your code base, as opposed to having a single file which clearly outlines the sequence of events that form the business process. Sounds great, am I missing anything here?
Correct me if I'm wrong, but your client publishes a command to RQM, server consumes it and then server publish an Event-response to RMQ which also will be handled asynchronously by the client. Isn't it a s a common event from choreography (EDA)? Because client doesn't wait for response from server. For example A sync. HTTP call from UI to client-microservice can't wait for result of such async response from client-microservice to server-microservice. And how will you synchronously response to UI about the result from response of server-microservice (except webSockets) ?
02:00 How about Transaction that you mention on the beginning, "inconsistent state"? When to persist transaction data? How to rollback the transaction? How to implement consistent state? This video does not yet answer these questions
I find myself still wondering about why the response would not be published as an event, for many or no consumers. In what scenarios might the described asynchronous Request-Response pattern be preferred over a similarly asynchronous Command-EventPublished pattern?
Good question. It's basically an reply to the sender of the result. There's a single reply for a single consumer as an indication of the completion. When in Pub/Sub, publishing an event isn't directly related to the invocation of a command. Meaning they aren't directly correlated (unless you made it be). Depending on the scenario you could be choose to listening to a topic, however that's not to say that event derived from the command/message, but that would be the case with a reply.
Hi @@CodeOpinion, I’ve been thinking about this problem too. Would it be possible and potentially ideal to mix the orch and choreography paradigms? You showed the Sales context trigger the process by handling the Bill-Order command but it could also do this by reacting to a Order-Created event through Pub/Sub. If you stick with pure orch. how are you able to prevent a process spanning potentially many parts of the business? Thanks for the vid!
I just rewatched the vid and realised my comment doesn’t actually make sense as Sales is the orchestrator. 😅 I think what I said still applies to other downstream cross-domain processes that may be triggered e.g. by an event within the Warehouse context, but not directly relevant to the goal of Bill-Order.
Yes you could use choreography instead. It's a trade off of centralizing the logic of workflow or not. Meaning billing could just consume the order placed event. However, as an example, not ever order needs to be billed. I'm this case, maybe being explicit in telling billing via a command (queued) could be more appropriate. As always, it really depends on the context.
Great video! Another thing we could consider is the unhappy path in these situations. What happens if an error happens in the process? Should we send domain errors as messages to the orchestrator for example? It seems that this way we could inform the user somehow (either in sync or async way).
Could you please do a video on how this plays out in a choreography scenario? and your opinion on how good is a choreography where the message has the process schema and each worker is responsible of either replying to the controller, or forwarding to the next worker.. thank you very much
Outbox is about reliably publishing events by persisting the events first in an atomic operation with the other state your changing. I talked about them in this video if you haven't watched already. ruclips.net/video/u8fOnxAxKHk/видео.html
It also worth to mention that you don't need to have event driven architecture or a broker for this pattern. You can send request to the server and immediately return a response with the job id and status while to server do the processing in the background.
I'm sorry for this noob question but i'm just starting on working as a .net developer. I need a bit of context for you videos that mention messaging, brokers, queues etc. When do i need these kind of implementations? When there are big applications separated in multiple services running independently? I think i have only dealt with monolith so this is why im confused. Stille I really appreciate ur videos.
Hi ! If you're just getting started, don't worry too much about architecture concerns for now or you'll quickly feel overwhelmed. This channel is great by usually covers quite advanced subjects. Most likely, these are decisions that will be made by senior team members / architects and are of a bigger scope. (unless you're the only member, and... well that's not great for a junior lol) You should focus more on what's relevant to your current project and try to improve bit by bit on smaller stuff. I really like this video + roadmap of Nick Chapsas to see what I could learn next : ruclips.net/video/gw-6lKrKlp0/видео.html
I’ve met a lots of people doing this but non which actually need it. Everyone things they need Netflix level solutions. Envelope it incrementally and make decisions as you go.
Azure Durable Functions is one of the best fits for these kind of scenarios with the power of Azure Storage Queues
For webjobs yeah but not for worker functions (isolated)
Webjobs is legacy name of functions. So what is your point!?
I agree with you, when I spent a lot of time targeting functions with isolated model ( no webjob dependencies) I don’t want to bring in webjobs to get the durables orchestrator to offload the work.
Lock vendor!
I would do an RPC call to the orchestrator, let it orchestrate everything, and then reply back. The downside is that the producer will be waiting. This approach is fine in a service as part of an asynchronous process in a service. Like within a consumer or separate process.
You should not do this call from a Web API in order to get a result back since it might take ages for responding back to the client. That defies the purpose.
In that case you would perform the RPC call and when you get the reply you will send it to the client using WebSockets.
I'm new to this topic, isn't the same problem present at both 2:06 and 4:38? I.e. in both situations, whether using synchronous or asynchronous request/response, your system will be left in an inconsistent state if the warehouse fails to respond? Is the benefit of asynchronous (e.g. using an orchestrator + message broker) purely just retry-logic?
Also what happens if the message broker goes down? Aren't you creating a critical dependency between all systems?
Also what is the lifecycle of the orchestrator? Is it an object that gets created for each new request and garbage collected when it receives all the responses it's expecting? What happens if one of your systems goes down and it starts creating heaps of orchestrators that fill up memory because they're not getting responses?
Sorry for the dumb questions, just trying to figure this all out.
Great set of questions.
While it seems like they are similar, when you're developing it to be asynchronous, there isn't any expectation of immediate consistency. In other words you know while the workflow is executing that time isn't apart of the mix. As your example if the warehouse isn't available, that doesn't mean the entire workflow fails. However if it were synchronous, if the warehouse weren't available the workflow/process does fail. Yes the retry logic is part of it but it's also removing that both need to be available at the same time. You can place an order and both billing and warehouse don't need to be available.
Yes the broker becomes important, but for reliability in not losing messages you'd use the outbox pattern: ruclips.net/video/u8fOnxAxKHk/видео.html
The orchestrator can contain state and depending on the library you're using will depend if it's in memory or getting created for each invocation. The alternative is event choreography, check out this video: ruclips.net/video/rO9BXsl4AMQ/видео.html
@@CodeOpinion okay so is the purpose of an orchestrator (over event choreography) purely to increase code readability and the ability to reason about your code base?
The implication being that it becomes really hard to mentally understand a business process by jumping through "hoops" of disparate events littered throughout your code base, as opposed to having a single file which clearly outlines the sequence of events that form the business process.
Sounds great, am I missing anything here?
Correct me if I'm wrong, but your client publishes a command to RQM, server consumes it and then server publish an Event-response to RMQ which also will be handled asynchronously by the client. Isn't it a s a common event from choreography (EDA)? Because client doesn't wait for response from server. For example A sync. HTTP call from UI to client-microservice can't wait for result of such async response from client-microservice to server-microservice. And how will you synchronously response to UI about the result from response of server-microservice (except webSockets) ?
02:00
How about Transaction that you mention on the beginning, "inconsistent state"? When to persist transaction data? How to rollback the transaction? How to implement consistent state?
This video does not yet answer these questions
Loved the pace
Appreciate the more complex topics that you cover in your channel.
Thanks for watching!
I find myself still wondering about why the response would not be published as an event, for many or no consumers. In what scenarios might the described asynchronous Request-Response pattern be preferred over a similarly asynchronous Command-EventPublished pattern?
Good question. It's basically an reply to the sender of the result. There's a single reply for a single consumer as an indication of the completion. When in Pub/Sub, publishing an event isn't directly related to the invocation of a command. Meaning they aren't directly correlated (unless you made it be). Depending on the scenario you could be choose to listening to a topic, however that's not to say that event derived from the command/message, but that would be the case with a reply.
Hi @@CodeOpinion, I’ve been thinking about this problem too. Would it be possible and potentially ideal to mix the orch and choreography paradigms? You showed the Sales context trigger the process by handling the Bill-Order command but it could also do this by reacting to a Order-Created event through Pub/Sub. If you stick with pure orch. how are you able to prevent a process spanning potentially many parts of the business?
Thanks for the vid!
I just rewatched the vid and realised my comment doesn’t actually make sense as Sales is the orchestrator. 😅 I think what I said still applies to other downstream cross-domain processes that may be triggered e.g. by an event within the Warehouse context, but not directly relevant to the goal of Bill-Order.
Yes you could use choreography instead. It's a trade off of centralizing the logic of workflow or not. Meaning billing could just consume the order placed event. However, as an example, not ever order needs to be billed. I'm this case, maybe being explicit in telling billing via a command (queued) could be more appropriate. As always, it really depends on the context.
Great video! Another thing we could consider is the unhappy path in these situations. What happens if an error happens in the process? Should we send domain errors as messages to the orchestrator for example? It seems that this way we could inform the user somehow (either in sync or async way).
Yes, the reply can be message that indicates an error or business process failure.
Could you please do a video on how this plays out in a choreography scenario? and your opinion on how good is a choreography where the message has the process schema and each worker is responsible of either replying to the controller, or forwarding to the next worker.. thank you very much
Yes, I have different videos illustrating this but I'll probably cover it specifically soon.
While watching this video (again), this pattern remind me the Transactional outbox pattern to make sure that message was processed. wdy?
Outbox is about reliably publishing events by persisting the events first in an atomic operation with the other state your changing. I talked about them in this video if you haven't watched already. ruclips.net/video/u8fOnxAxKHk/видео.html
Great video! very important topic
It also worth to mention that you don't need to have event driven architecture or a broker for this pattern.
You can send request to the server and immediately return a response with the job id and status while to server do the processing in the background.
True, however if you need reliability you want the message to be durable.
@@CodeOpinion Right. I really appreciate that you find time to replyand give your feedback :)
I'm sorry for this noob question but i'm just starting on working as a .net developer. I need a bit of context for you videos that mention messaging, brokers, queues etc.
When do i need these kind of implementations? When there are big applications separated in multiple services running independently?
I think i have only dealt with monolith so this is why im confused.
Stille I really appreciate ur videos.
Hi ! If you're just getting started, don't worry too much about architecture concerns for now or you'll quickly feel overwhelmed. This channel is great by usually covers quite advanced subjects.
Most likely, these are decisions that will be made by senior team members / architects and are of a bigger scope. (unless you're the only member, and... well that's not great for a junior lol)
You should focus more on what's relevant to your current project and try to improve bit by bit on smaller stuff.
I really like this video + roadmap of Nick Chapsas to see what I could learn next : ruclips.net/video/gw-6lKrKlp0/видео.html
I’ve met a lots of people doing this but non which actually need it. Everyone things they need Netflix level solutions. Envelope it incrementally and make decisions as you go.