How can I prevent an increasing backlog of messages in my Amazon SQS queue?

Short description

Standard and FIFO SQS queues

The backlog of messages increases under the following circumstances:

• Producers send messages at a faster rate than the messages are consumed.
• Consumers dont delete messages within the visibility timeout period. When the SQS queue is polled, messages reappear in the queue.

FIFO SQS queues

For FIFO (First-In-First-Out) SQS queues, an increase in the backlog of messages can also result from either of the following:

• The FIFO queue 20,000 message buffer limit.
• A message that belongs to a message group is stuck with a consumer, preventing other messages from the same message group from processing.

Resolution

Follow these best practices to prevent the backlog of messages from increasing.

Standard and FIFO SQS queues

• Set the SQS queue optimum visibility timeout to allow the consumer to delete messages after processing them within the visibility timeout period. If you dont know how long it takes to process a message, then create a heartbeat for your consumer process. Specify the initial visibility timeout (for example, 2 minutes). Then, if the consumer needs more time to process the message, continue to increase the visibility timeout using the ChangeMessageVisibility API call.
• Increase the batch size when you make ReceiveMessage API calls. Set the MaxNumberOfMessages parameter value to more than 1 and up to a maximum of 10.
• Monitor the SQS queue metric Approximate Number of Messages Visible. This metric allows you to see if producers start to produce messages at a higher rate than consumers can consume the messages. To scale horizontally, increase the number of consumers or clients that consume the SQS queue, or increase the number of threads that poll the queue.

FIFO SQS queues

20,000 message buffer

FIFO queues allow a maximum of 20,000 inflight messages. Inflight messages include those received from a queue by a consumer, but not yet deleted from the queue. If you reach the 20,000 quota, Amazon SQS doesnt return error messages. A FIFO queue looks through the first 20,000 messages to determine available message groups. If you have a backlog of messages in a single message group, you cant consume messages from other message groups until you successfully consume messages from the backlog.

Scaling over message groups

Messages that belong to the same message group are processed one by one, in the order relative to the message group. When receiving messages with multiple message group IDs, Amazon SQS tries first to return as many messages with the same message group ID as possible. This allows other consumers to process messages with a different message group ID. When messages that belong to a specific message group ID are invisible, no other consumer can process messages with the same message group ID. However, consumers can process messages from other message groups. Try increasing the number of message groups where the order isnt important.