Here is a blog post that goes through how you can build a streaming data backend by using a serverless approach. The blog provides a complete example application that you can deploy to your AWS account: aws.amazon.com/blogs/compute/building-serverless-applications-with-streaming-data-part-1/
Really I don't understand how a person who produce such a great quality content can have only 10k subs. Let's keep up this way, you're doing a great job.
Thank you brother! My content is very niched, and I like it this way. I can produce videos that go over the top to get more views, but it's not my focus right now, maybe one day
Thanks! Really interesting pattern. Been looking at event sourcing patterns for awhile, and this is one of the most immediately practical videos I've seen on the topic. Looking forward to part 2.
Amazing content!!! Thanks for your work. Please We need more use cases, like examples of scenarios and your proposed architectures and patterns applied. Thanks in advance.
I've been getting confused with the different Amazon Kinesis products- so I'm glad they were used in this architectural design pattern. I have a feeling this channel will get much bigger. Keep up the good work.
At 11:36, you have an SQS queue between your consumer and producer lambdas. Why not just have a single lambda that reads from one kinesis data stream to the other? I'd imagine that the SQS queue would largely bring disadvantages. For example, if you use a regular (non-FIFO) sqs queue, then you lose the ordering you had within each kinesis data stream shard. If, instead, you use a FIFO queue, then you potentially create a bottleneck due to its lower throughput per message group (even if you, say, set the message group id to equal the shard id and, especially, if batching is enabled for kinesis data stream). Also, SQS has a smaller message size limit (256kb vs 1MB I wanna say?), so, it might be a no-go to begin with.
The SQS queue is mainly there to act like a buffer while the producer reads the messages from SQS in a batch manner. The idea is to leverage Lambda concurrency model + Lambda batch processing capabilities to save on calls cost: For example, we can configure Lambda to wait until 10 messages are in the queue (or once every 30 sec) before doing one round trip to SQS. This doesn't only save on cost, but also scales better since you most likely won't go over the Lambda invocation limit your account have. As for the message size limit, of course a bigger size message would be a new constraint that will most likely require a design change, for example: Consumer B puts object to S3, then Producer reads that object...
@@WinTheCloud Even though kinesis data stream works just fine as a buffer? My thoughts: Just have the kinesis data stream / lambda event source be configured with a batch size of, say, 200 records and a maximum wait time of 60 seconds. Your lambda would then poll for either 200 records or until 60 seconds has passed (whichever occurs first) before processing them all (and pushing the processed records to that other fraud detection kinesis data stream) at once. Far as I can tell, this would provide the same buffering effect you're going for. If you further keep the ParallelizationFactor at 1 (i think this is the default value), then you'd keep the within-shard ordering that you'd lose w/ a non-FIFO SQS queue. You'd end up with far fewer lambda invocations (as lambda can poll up to 1,000 records from kinesis vs 10 from SQS), fewer components (1 sqs queue and 1 lambda fewer), and fewer added constraints (kinesis data stream record size limit).
@Robin Chan Yes, but you get the same benefits (and more) by removing the SQS queue there. I like the architecture design (with the SQS queue removed) and agree with your comment that the main focus was on the separation of ownership. It's just that the SQS queue there doesn't help anything and yet adds size & ordering limitations. So, I'm saying remove the SQS queue and keep the rest.
IMO the most important part of serverless is, that you do not need to select any size, server model etc. when provisioning a service. Therefore Kinesis is managed, but not serverless as you need to select the shard count.
Very informative! Can you please do a architectural design for serverless AWS that can communicate to other APIs like Stripe? Thanks for the awesome videos.
I haven't seen many cases where optimistic locking is handled on the data stream itself. Some AWS services like DynamoDB and ElasticSearch are pretty good at this: docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html
I can’t thank you enough for your vids! They help me prepare for everything from coding to cloud certifications and best practices from all over the cloud industry. Plus your approach to teaching it is so well presented and easy to understand!!
Yes, definitely, the SQS is there to add a buffer in case the load spikes, but also to avoid tight coupling: If you decide to use Kafka, later rather than Kinesis, you won't have to change your Lambda function code ;)
What happens if as an example, from one domain to the other, I want to send some information that is under GDPR protection, so it needs to be deleted at some point. Can I set up a lambda between Data streams and Firehose such that I strip that data from the event and only use it to produce a report (let's say) but not have it permanently stored in the S3 bucket?
Here is a blog post that goes through how you can build a streaming data backend by using a serverless approach. The blog provides a complete example application that you can deploy to your AWS account: aws.amazon.com/blogs/compute/building-serverless-applications-with-streaming-data-part-1/
I would like to ask you, what online certified courses would you recommend for becoming a cloud engineer?
@@RaGePr00fs I have dedicated a video for this specific question ;) ruclips.net/video/-eRfvkfCY0Y/видео.html
@@WinTheCloud Ohh :DDD thank you. I watched after I wrote the comment back then ) . Thank you for your hard -work and following up.
Man, thank you! I've been watching your videos for 5 days and gonna watch all of them!
Awesome, thank you! I'm really glad you like them. I was starting to get worried people didn't find them useful..
Really I don't understand how a person who produce such a great quality content can have only 10k subs.
Let's keep up this way, you're doing a great job.
Thank you brother! My content is very niched, and I like it this way. I can produce videos that go over the top to get more views, but it's not my focus right now, maybe one day
keep posting educative videos like this man
As long as you keep giving them likes haha ! Jokes aside, I'll definitely do my best!
Thanks! Really interesting pattern. Been looking at event sourcing patterns for awhile, and this is one of the most immediately practical videos I've seen on the topic. Looking forward to part 2.
Great video! I'm surprised that you don't have more subscribers.
Thank you my friend! if you like the video, please give it a like, apparently it helps the channel to be more suggested to other people
Amazing Video... excited about upcoming videos !!!
More to come!
Amazing content!!! Thanks for your work. Please We need more use cases, like examples of scenarios and your proposed architectures and patterns applied. Thanks in advance.
Thank you, Luis! I have more patterns in store, next I'm going to be talking about the GateKeeper pattern and more in the same style.
@@WinTheCloud 😊👍
I've been getting confused with the different Amazon Kinesis products- so I'm glad they were used in this architectural design pattern.
I have a feeling this channel will get much bigger. Keep up the good work.
Glad it was helpful!
Exactly what I was looking for
Glad it was helpful Shadid ;)
i'm currently studying to get my solutions architect certificate (associate level). your videos are really great. thanks
Super happy to hear it! Good luck for your certificate
@@WinTheCloudThanks
alot to learn from you thanks, buddy!!
At 11:36, you have an SQS queue between your consumer and producer lambdas. Why not just have a single lambda that reads from one kinesis data stream to the other? I'd imagine that the SQS queue would largely bring disadvantages.
For example, if you use a regular (non-FIFO) sqs queue, then you lose the ordering you had within each kinesis data stream shard. If, instead, you use a FIFO queue, then you potentially create a bottleneck due to its lower throughput per message group (even if you, say, set the message group id to equal the shard id and, especially, if batching is enabled for kinesis data stream).
Also, SQS has a smaller message size limit (256kb vs 1MB I wanna say?), so, it might be a no-go to begin with.
The SQS queue is mainly there to act like a buffer while the producer reads the messages from SQS in a batch manner. The idea is to leverage Lambda concurrency model + Lambda batch processing capabilities to save on calls cost: For example, we can configure Lambda to wait until 10 messages are in the queue (or once every 30 sec) before doing one round trip to SQS. This doesn't only save on cost, but also scales better since you most likely won't go over the Lambda invocation limit your account have.
As for the message size limit, of course a bigger size message would be a new constraint that will most likely require a design change, for example: Consumer B puts object to S3, then Producer reads that object...
@@WinTheCloud Even though kinesis data stream works just fine as a buffer?
My thoughts: Just have the kinesis data stream / lambda event source be configured with a batch size of, say, 200 records and a maximum wait time of 60 seconds. Your lambda would then poll for either 200 records or until 60 seconds has passed (whichever occurs first) before processing them all (and pushing the processed records to that other fraud detection kinesis data stream) at once. Far as I can tell, this would provide the same buffering effect you're going for. If you further keep the ParallelizationFactor at 1 (i think this is the default value), then you'd keep the within-shard ordering that you'd lose w/ a non-FIFO SQS queue. You'd end up with far fewer lambda invocations (as lambda can poll up to 1,000 records from kinesis vs 10 from SQS), fewer components (1 sqs queue and 1 lambda fewer), and fewer added constraints (kinesis data stream record size limit).
@Robin Chan Yes, but you get the same benefits (and more) by removing the SQS queue there.
I like the architecture design (with the SQS queue removed) and agree with your comment that the main focus was on the separation of ownership.
It's just that the SQS queue there doesn't help anything and yet adds size & ordering limitations. So, I'm saying remove the SQS queue and keep the rest.
@@somedude31 The benefit of the queue is that any number of other subscribers can subscribe to the queue as the application grows.
I think there are some major misunderstandings about event-sourcing here- this is an event-driven pattern, not event-sourcing.
IMO the most important part of serverless is, that you do not need to select any size, server model etc. when provisioning a service. Therefore Kinesis is managed, but not serverless as you need to select the shard count.
I see your point. I use the same definition for Serverless as well (no size, mode... are required)
Very informative! Can you please do a architectural design for serverless AWS that can communicate to other APIs like Stripe? Thanks for the awesome videos.
How about this one? ruclips.net/video/yBGLw_eITfY/видео.html Let me know if you have questions!
Interesting video. How does Kinesis help you handling optimistic concurrency exceptions? That is a must to validate commands before emitting events
I haven't seen many cases where optimistic locking is handled on the data stream itself. Some AWS services like DynamoDB and ElasticSearch are pretty good at this: docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html
Good One.
Thank you! Cheers!
Great!
Thank you. I'm happy you enjoyed it
Audio quality is great.
Thank you for the feedback! I appreciate it
Audio sounds great! Amazing video too!
Much appreciated!
I can’t thank you enough for your vids! They help me prepare for everything from coding to cloud certifications and best practices from all over the cloud industry. Plus your approach to teaching it is so well presented and easy to understand!!
Nice solution. Just to be safe: at 9:12 that DynamoDB table is just a denormalized view of the events?
It is. This DynamoDB table can be your main DB that your app reads from, so you can use whatever schema your app requires
Nice video. Can we push messages directly from lambda to kinesis to avoid sqs in this solution?.
Yes, definitely, the SQS is there to add a buffer in case the load spikes, but also to avoid tight coupling: If you decide to use Kafka, later rather than Kinesis, you won't have to change your Lambda function code ;)
What happens if as an example, from one domain to the other, I want to send some information that is under GDPR protection, so it needs to be deleted at some point. Can I set up a lambda between Data streams and Firehose such that I strip that data from the event and only use it to produce a report (let's say) but not have it permanently stored in the S3 bucket?
lol can u guide road map to solution architect
Why not eventbridge?
EventBridge works for this as well - You can also use SQS to sort of "stream" data rather than Kinesis Data Stream... Many variations are possible