Good question - I might have tried to tackle too much with this one :D In order to make your application highly available, you need to have processing in multiple regions. In theory, if one region goes down, you should be able to switch to the other region no problem. SQS doesn't really have a way to handle this natively, but you can handle it by publishing and processing the messages in multiple regions using SNS as a global publisher and SQS in each region. With this comes the problem of duplication - now you are processing the message multiple times in multiple regions. Even with this, SQS also does not guarantee that a message is not processed only once with standard queues - this makes it a problem even if you are not doing multiple regions. (note - you could use an SQS FIFO queue instead, but that can come with other trade-offs) To fix the problem of duplication, you could make the processing logic idempotent, meaning that even if a message is processed multiple times in different regions, the result remains consistent. This basically makes the duplication issue a non-problem because the result will always be the same. Another option is to store the message in a global database (like with global tables in Dynamo DB) to track and deduplicate them across regions. This is a more likely approach, but you could run into race conditions so you might add a delay somewhere to avoid that. I hope this helps! Let me know if you have more questions.
always nice to see that youre still active around here
Trying my best :)
Yayy she’s back!!!
Hoping to post more content next year
Hi!
3:13 how replication solves a problem of duplication? I didn't get it.
Good question - I might have tried to tackle too much with this one :D
In order to make your application highly available, you need to have processing in multiple regions. In theory, if one region goes down, you should be able to switch to the other region no problem. SQS doesn't really have a way to handle this natively, but you can handle it by publishing and processing the messages in multiple regions using SNS as a global publisher and SQS in each region.
With this comes the problem of duplication - now you are processing the message multiple times in multiple regions. Even with this, SQS also does not guarantee that a message is not processed only once with standard queues - this makes it a problem even if you are not doing multiple regions. (note - you could use an SQS FIFO queue instead, but that can come with other trade-offs)
To fix the problem of duplication, you could make the processing logic idempotent, meaning that even if a message is processed multiple times in different regions, the result remains consistent. This basically makes the duplication issue a non-problem because the result will always be the same.
Another option is to store the message in a global database (like with global tables in Dynamo DB) to track and deduplicate them across regions. This is a more likely approach, but you could run into race conditions so you might add a delay somewhere to avoid that.
I hope this helps! Let me know if you have more questions.