There's a very good article about it called "Choreography vs Orchestration in the land of serverless", TL;DR: Use both in the same project, using orchestrators inside a bounded context and choreography between bounded contexts.
so glad I found this channel, much appreciate your effort. I am just learning AWS and this mthod of teaching find is so much easier to understand. Whereas others will just mention Services in silos and then start drilling down into all the granual info on each Service (without even explaining how it relates/connects to other services (making it difficult to get your head around it all).
We heavily use Step Functions at work. We don't require a rollback in the event a problem happens but we do need to know that a failure happened. For that we use Cloudwatch integration with slack that monitors each lambda in the workflow. Note that in each step function task that executes a lambda, if mandated you could essentially add the retries, exponential backoff and trigger details as well to reprocess the lambda and perhaps move it into a DLQ for redriving it at a later time. BTW, thank you for all the AWS content and easy explanation that you provide. I've benefited a lot from it.
There is another option if you are using Standard step functions. You could create an Event Bus rule that reacts to Step Function transition to Failed and as an action it can publish the message to SNS and from SNS you can send it everywhere. But this is an option only if you use Standard. For Express I usually use the Async invocation and let the caller deal with the potential failure of the Step function.
@@cristiannechita2032 Is there a benefit to doing that using event bus instead of having the step function routing the error message and input payload to say another step for SNS in the step function itself? I've tested this and it works well when a lambda fails and the retries are exhausted since step functions are async operations.
@@arkster00 The benefit that I see is that you keep your step function cleaner (without putting those transition from various nodes towards a common error handling task). Also, there are Tasks without Lambdas (e.g. Pass) that can apply transformation and those can fail as well. This should not replace the retry operations of your lambda step (it makes sense to keep it to deal with any transient errors).
This really brings back memories of when I used to use OOP in C++ to create an event driven application. Thanks! Now, let me implement this in OOP Python first.
great video. it reminds me of the difference between OOP (mapped to EDA) vs non-OOP (mapped to workflow). Solving the same problem but modeling the problem differently which leads to various tradeoffs
Hi, Thanks for this video. I would like to know why we cannot maintain microservice architecture that are handled by individual small team while using step functions? Still the step functions can be created or maintained by the architect and we can have different teams to maintain the services right? Thanks.
Look at the step-function as being part of a micro-service. That team owns everything regarding that workflow (from it's definition in IaC till deployment).
So basically if my application has 3 services working independently, i can assume that those services can be monitored separately using step functions right ? Its to say, micro monitoring sortof a way, eh ?
I am not sure what is happening between the two lambda functions that can determine the success/timeout/default of the first Choice node. An iot device? Like a sensor? Is it forwarding data? What reads that data?
wouldn't you have to revert changes made in a step in case of failure when using step function approach, or you would simply correct the failing part i.e ask customer for another credit/debit card and retry? In my experience only durable functions on azure have the functionality to re-submit the request, i am sure there would be something similar on AWS end.
Well it depends on where you redirect the customer. If you want your customers to pay with another channel then you can build your logic in triggering another lambda and at the same time route the user on the frontend of payments channels list, so that user can select the payment channel of his choice. Auto resubmission will make user stuck in the loop.
Step functions are great, but I’ve found monitoring to be a weak spot. The state display on the diagrams is very rudimentary, and you’ve got to use other tools and work with the logs to actually get a real monitoring dashboard.
Thanks for the video and example, learned new kinds of stuff on EDA and workflow. I was thinking, in the case of the EDA approach again for monitoring, maybe we should log or publish the state and the success/failure of every operation. Triggering a lambda on failed operation/step could also help us in unwinding initial steps and marking all of them as invalid. Does it make sense?
There are so many more sub-events in the logic and execution in handling the transaction steps in a distributed system just for placing the order based on the user's shopping cart that you gloss over. That would be enough for a single video alone. The credit card service would be inside the order placement, just for starters! Before doing any subsequent fulfillment steps let alone packaging, a hold would need to be placed for the required funds on the user's account with the card issuer 😂 You need something like Kafka/Kinesis as an event log to distribute the events to appropriate consumers, etc etc. I do agree that the Workflow approach has significant benefits, and you are right about the costs and tradeoffs of orchestration with this approach. I just think that you need to go into a bit more detail with the event-based architecture side, because your diagram is extremely oversimplified. And I don't think IoT is an essential aspect of the system at all, these sorts of systems functioned before such devices even existed.
Good video. Reality is fancy bla bla architects see the first diagram on event driven choreography as a short running stateful work flow just because there are arrows in diagram & then bar of argumentation starts with integration SMEs 😂😂
Great content, I've been watching you for a few months. I wanted to give feedback on the timing at the end of your videos with suggestions. Within a few seconds of suggesting other videos and overlaying them to be clicked, the video ends and your suggestions are no longer available. It would be nice to have a bit more time to react: read the video titles, determine the relevancy to what I am trying to learn, and then clicking.
While I love this vid, I think @sarc007 has a point. This is almost never the case in reality. Companies want your money ASAP and will confirm pre-auths in a nightly batch, or up to 3 days later. If your shipping flow gets stalled, you don't want your pre-auths timing out. It feels like you used artistic license to switch them around so you could make your point about observability.@@BeABetterDev
Not totally true😊 In the Eda design you are using SNS and it will generate chatty conversation between teams And if we focus on microservice only following the essence of DDD Each step in Workflow could be a service so just handling service with well defined bounded context will fit team responsibilities very well
Love the way you explain things. Very clear and concise. Great work!
Be nice if channel owner just mentioned a quick appreciation note for your generosity …
There's a very good article about it called "Choreography vs Orchestration in the land of serverless", TL;DR: Use both in the same project, using orchestrators inside a bounded context and choreography between bounded contexts.
Good article!
Did the author said how could define the boundary? AKA when to use orchestrator and when to use choreography
I was wondering if he would end on that note. Seems like together they would counter each other's cons.
so glad I found this channel, much appreciate your effort. I am just learning AWS and this mthod of teaching find is so much easier to understand. Whereas others will just mention Services in silos and then start drilling down into all the granual info on each Service (without even explaining how it relates/connects to other services (making it difficult to get your head around it all).
You're very welcome and welcome!
With event driven you could use traces for observability to visualize what step you’re at.
We heavily use Step Functions at work. We don't require a rollback in the event a problem happens but we do need to know that a failure happened. For that we use Cloudwatch integration with slack that monitors each lambda in the workflow. Note that in each step function task that executes a lambda, if mandated you could essentially add the retries, exponential backoff and trigger details as well to reprocess the lambda and perhaps move it into a DLQ for redriving it at a later time.
BTW, thank you for all the AWS content and easy explanation that you provide. I've benefited a lot from it.
There is another option if you are using Standard step functions. You could create an Event Bus rule that reacts to Step Function transition to Failed and as an action it can publish the message to SNS and from SNS you can send it everywhere. But this is an option only if you use Standard. For Express I usually use the Async invocation and let the caller deal with the potential failure of the Step function.
@@cristiannechita2032 Is there a benefit to doing that using event bus instead of having the step function routing the error message and input payload to say another step for SNS in the step function itself? I've tested this and it works well when a lambda fails and the retries are exhausted since step functions are async operations.
@@arkster00 The benefit that I see is that you keep your step function cleaner (without putting those transition from various nodes towards a common error handling task). Also, there are Tasks without Lambdas (e.g. Pass) that can apply transformation and those can fail as well.
This should not replace the retry operations of your lambda step (it makes sense to keep it to deal with any transient errors).
@@cristiannechita2032 Thanks. Something to think about.
This channel is gold ❤️
As always, this video covers everything in a comprehensive manner and makes it easy to apply practically
This is a great video, and can be greater when adding the monolith client retrying approach because tons of people are still stuck in there.
This really brings back memories of when I used to use OOP in C++ to create an event driven application. Thanks! Now, let me implement this in OOP Python first.
Informative! What s/w have you used to show the workflow?
great video. it reminds me of the difference between OOP (mapped to EDA) vs non-OOP (mapped to workflow). Solving the same problem but modeling the problem differently which leads to various tradeoffs
Wow. This is what I just said.
Hi what do you think about using SQS + Dead letter queue in event-driven architecture to help monitor
@Be A Better Dev can't you use Distributed Tracing to see status of orders in SOA?
Thank you very much for your explanation, which is nice and easy to understand. 👍
Fantastic video! Extremely clear and thorough on both patterns
Thank you for a great presentation
Glad you enjoyed it!
Hi, Thanks for this video.
I would like to know why we cannot maintain microservice architecture that are handled by individual small team while using step functions?
Still the step functions can be created or maintained by the architect and we can have different teams to maintain the services right?
Thanks.
Look at the step-function as being part of a micro-service. That team owns everything regarding that workflow (from it's definition in IaC till deployment).
Christian's point is spot-on.
So basically if my application has 3 services working independently, i can assume that those services can be monitored separately using step functions right ?
Its to say, micro monitoring sortof a way, eh ?
Got it. Thanks Cristian.
I am not sure what is happening between the two lambda functions that can determine the success/timeout/default of the first Choice node. An iot device? Like a sensor? Is it forwarding data? What reads that data?
wouldn't you have to revert changes made in a step in case of failure when using step function approach, or you would simply correct the failing part i.e ask customer for another credit/debit card and retry? In my experience only durable functions on azure have the functionality to re-submit the request, i am sure there would be something similar on AWS end.
Well it depends on where you redirect the customer. If you want your customers to pay with another channel then you can build your logic in triggering another lambda and at the same time route the user on the frontend of payments channels list, so that user can select the payment channel of his choice. Auto resubmission will make user stuck in the loop.
Step functions are great, but I’ve found monitoring to be a weak spot. The state display on the diagrams is very rudimentary, and you’ve got to use other tools and work with the logs to actually get a real monitoring dashboard.
what will be the best template in lucid chart to replicate the step function workflow part?
Good Explanation. Thanks
Thank you so much for the explanation.. pleas can you do a video more on Aws IOT service and also the more on database
Thanks for the video and example, learned new kinds of stuff on EDA and workflow.
I was thinking, in the case of the EDA approach again for monitoring, maybe we should log or publish the state and the success/failure of every operation. Triggering a lambda on failed operation/step could also help us in unwinding initial steps and marking all of them as invalid. Does it make sense?
Great job man. Thanks
What are you doing step-function?
Great lecture!! Which tool do you use to draw a visual representation of architecture ?
Thanks Eduard! I used Google slides for the first part, and the Step Functions Studio Editor for the second half. Cheers.
Very helpful!
Thanks for sharing this information, useful 👍
you are a super hero. thanks a looot
another great video as always!
Glad you enjoyed!
Great explanation!
can you do workflow withut step functions?
What do you use to draw the illustration? 9:20
I'm almost sure this is the editor for StateMachine (a.k.a StepFunctions) in AWS.
There are so many more sub-events in the logic and execution in handling the transaction steps in a distributed system just for placing the order based on the user's shopping cart that you gloss over. That would be enough for a single video alone. The credit card service would be inside the order placement, just for starters! Before doing any subsequent fulfillment steps let alone packaging, a hold would need to be placed for the required funds on the user's account with the card issuer 😂
You need something like Kafka/Kinesis as an event log to distribute the events to appropriate consumers, etc etc.
I do agree that the Workflow approach has significant benefits, and you are right about the costs and tradeoffs of orchestration with this approach.
I just think that you need to go into a bit more detail with the event-based architecture side, because your diagram is extremely oversimplified. And I don't think IoT is an essential aspect of the system at all, these sorts of systems functioned before such devices even existed.
Thanks
Awesome video
Looks like absolute overkill architecture for 99,9999 % of all apps and businesses 🤔. But, good explanation and overview of AWS madness. 😊
very helpful video
good video
Glad you enjoyed!
Good video. Reality is fancy bla bla architects see the first diagram on event driven choreography as a short running stateful work flow just because there are arrows in diagram & then bar of argumentation starts with integration SMEs 😂😂
Great content, I've been watching you for a few months.
I wanted to give feedback on the timing at the end of your videos with suggestions. Within a few seconds of suggesting other videos and overlaying them to be clicked, the video ends and your suggestions are no longer available. It would be nice to have a bit more time to react: read the video titles, determine the relevancy to what I am trying to learn, and then clicking.
Workflow is Event-driven with global state.
Nice
all workеd
👍💯
Customer Charging happens first and rest other events follow
Not always. Cards are usually pre-authorized for a charge and charged only upon shipping.
While I love this vid, I think @sarc007 has a point. This is almost never the case in reality. Companies want your money ASAP and will confirm pre-auths in a nightly batch, or up to 3 days later. If your shipping flow gets stalled, you don't want your pre-auths timing out. It feels like you used artistic license to switch them around so you could make your point about observability.@@BeABetterDev
Not totally true😊
In the Eda design you are using SNS and it will generate chatty conversation between teams
And if we focus on microservice only following the essence of DDD
Each step in Workflow could be a service so just handling service with well defined bounded context will fit team responsibilities very well
well, good video but seems this is not event-driven.
Thanks