amazon-s3 - Using S3 to group SQS messages instead of FIFO queues

Question

We plan to use SQS standard queue for messaging. The messages are a bunch of records that need to be received in order at the consumer end. There are a few reasons due to which we plan not to use FIFO queues.

1. FIFO queues have a few restrictions as noted here and are recommended only for few selected use cases.
2. We have multiple producers that push such messages to the queue (all producers are independent of the other) hence we are most likely to hit the 300 messages per second limitation.

Given this we are evaluating the SQS extended library support to use S3 for storing message payloads. We would club all the linked records into one message and post it to SQS as one request. I have a few questions

1. What are the limitations or side effects of using S3 for persisting message payloads? One that I am aware off would be - S3 cost - I am assuming this won't be large given our messages don't cross a few MB's max.
2. Are there real world examples of using this approach over FIFO queues for grouping messages?

标签： amazon-s3amazon-sqsfifo

Answer 1

S3 introduces additional latency (at each end of the queue), depending on the payload size, whether the message publishers/consumers are in AWS or hosted elsewhere, and how much bandwidth an individual server instance has available to it. (Off the cuff, I’d guess it’s >200 ms for a 1 MB payload.) The cost will be pretty insignificant, especially if you set up appropriate bucket lifecycle policies to archive or delete old data. Don’t forget that S3 is strongly consistent on the initial create, but only eventually consistent for any updates to an object. If possible, don’t update an object once you have created it.

I don’t have any real world examples, but I’ll let you know if I find one.

You will probably find it simpler to implement what you need using some sort of database, as was suggested in the article that you linked (which explained the limitations of FIFO queues). Make sure your decision isn’t biased by looking for premature optimizations.