SqsListener dequeues are incorrectly throttled

[JasonGrundy]

I have used Spring Boot and Spring Cloud AWS for many years. In July I upgraded to 3.5.0 and 3.4.0 respectively.

Everything worked well until this month. However we have started to observe an issue with one on the listeners (all the other ones are OK). The volume of messages in the underlying SQS is not very large (1-5k per hour).

What happens is that after a period it flatlines at 2.5k dequeues per hour and is never able to process more messages than this. This has happened 4 times. If we restart the component (it's an ECS Task) then it behaves normally and quickly clears the queue. The problem is not in the code that processes the queue items. We can see that for each batch of messages we quickly process them and then wait 10 seconds before initiating a further dequeue. We're using the default configuration for everything. We cannot replicate this problem in a lower environment (can take days to occur in production)

We are about to enable additional logging and are hopeful that this sheds light on what is happening (DEBUG for io.awspring.cloud.sqs). My question is whether anyone has other suggestions about how to troubleshoot this type of issue.

[tomazfernandes]

Hey @JasonGrundy, thanks for bringing this up.

This section of the documentation explains some of the throughput configuration options.

For instance, the default pollTimeout is 10s, which means the listener will wait until that time to try to fetch a full batch of 10 messages. That'd be consistent with the 2.5k polls / hour you're experiencing.

You could try:

• Decreasing pollTimeout (the tradeoff is potentially more poll requests to SQS - more cost)
• Decreasing maxMessagesPerPoll (same tradeoff with cost)
• Increasing maxConcurrentMessages if you need faster processing when the queue has more messages.
• Setting the BackPressureMode to FIXED_HIGH_THROUGHPUT so it doesn't switch to low mode.

It'd also be helpful for to analyze what's the message production pattern for this particular queue in contrast with the other queues.

Let me know if that helps, thanks.