SQS Queue Should Not Have A High Number of Unprocessed Messages

More Info:

Amazon Simple Queue Service (SQS) queues should not be holding a high number of unsuccessfully-processed messages due to unresponsive or incapacitated consumers.

Risk Level

Medium

Address

Reliability

Compliance Standards

• APRA CPS 234 (Australia)
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIS2 Directive
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• PCI
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Using Console

Step 1: Log into the AWS Management Console

Step 2: Navigate to the Amazon Simple Queue Service (SQS) dashboard by selecting ‘Services’ from the top menu and then selecting ‘SQS’ under the ‘Application Integration’ section.

Step 3: In the Amazon SQS dashboard, you will see a list of all your SQS queues. Identify the queue that has a high number of unprocessed messages.

Step 4: Click on the queue name to open its details.

Step 5: Check the ‘Messages Available for Processing’ metric. If this number is high, it means that there are many unprocessed messages in the queue.

Step 6: To remediate this, you need to either increase the number of consumers processing messages from the queue or increase the processing speed of the existing consumers.

To increase the number of consumers:

Step 7: Go to the AWS Lambda console and find the function that is processing the messages from the SQS queue.

Step 8: Increase the ‘Concurrent executions’ limit for the Lambda function. This will allow more instances of the function to run in parallel, thus processing messages from the queue faster.

To increase the processing speed of the existing consumers:

Step 9: Review the code of the Lambda function that is processing the messages. Look for any inefficiencies or bottlenecks that could be slowing down the processing speed.

Step 10: Optimize the code to process messages faster. This could involve things like batching operations, using faster algorithms, or reducing the amount of data being processed.

Step 11: After making changes, monitor the ‘Messages Available for Processing’ metric in the SQS dashboard to ensure that the number of unprocessed messages is decreasing.

Remember, the goal is to ensure that messages are being processed at least as fast as they are being added to the queue. If the number of unprocessed messages continues to grow, you may need to further increase the number of consumers or the processing speed.

Using CLI

In order to remediate this issue, you need to identify the reason behind the high number of unprocessed messages in your SQS queue. The problem could be due to slower processing of messages by the consumer or the producer sending messages at a faster rate than the consumer can handle.

Here are the steps to remediate this issue:

1. Identify the Queue: Use the following AWS CLI command to list all your SQS queues and identify the one with the high number of unprocessed messages:

   aws sqs list-queues
   

2. Check the Queue Attributes: Use the following AWS CLI command to check the attributes of the identified queue:

   aws sqs get-queue-attributes --queue-url <Your-Queue-URL> --attribute-names All
   

   Look for the ApproximateNumberOfMessages and ApproximateNumberOfMessagesNotVisible attributes. These will give you an idea of the number of messages waiting to be processed and the number of messages currently being processed.

3. Identify the Issue: If the number of ApproximateNumberOfMessages is high, it means your consumer is not processing the messages fast enough. If the ApproximateNumberOfMessagesNotVisible is high, it means the messages are not being deleted after being processed.

4. Increase the Number of Consumers: If the issue is with the consumer not processing the messages fast enough, you may need to increase the number of consumers. This can be done by either increasing the number of instances of your consumer application or by increasing the number of threads in your consumer application that are processing the messages.

5. Increase the Visibility Timeout: If the issue is with the messages not being deleted after being processed, you may need to increase the visibility timeout. This can be done using the following AWS CLI command:

   aws sqs set-queue-attributes --queue-url <Your-Queue-URL> --attributes VisibilityTimeout=<New-Timeout-Value>
   

6. Monitor the Queue: After making these changes, monitor the queue to see if the number of unprocessed messages decreases. If it doesn’t, you may need to further increase the number of consumers or the visibility timeout.

Remember to replace <Your-Queue-URL> and <New-Timeout-Value> with your actual queue URL and desired timeout value.

Using Python

You can use the Boto3 library in Python to interact with AWS services including SQS. Here’s how you can remediate the issue:

1. Install Boto3: You can install Boto3 using pip. If you haven’t installed it yet, you can do so using the following command:

   pip install boto3
   

2. Set up AWS credentials: You need to configure your AWS credentials. You can do this by using the AWS CLI and running aws configure. This will prompt you for your AWS Access Key ID, Secret Access Key, default region name, and default output format.

3. Import Boto3: In your Python script, you need to import the Boto3 library. You can do this with the following line of code:

   import boto3
   

4. Create SQS Client: You need to create an SQS client. You can do this with the following line of code:

   sqs = boto3.client('sqs')
   

5. Get the Queue URL: You need to get the URL of the SQS queue. You can do this with the following line of code:

   queue_url = sqs.get_queue_url(QueueName='YOUR_QUEUE_NAME')['QueueUrl']
   

6. Process the Messages: Now, you need to process the messages. You can do this with a while loop that continues to run as long as there are messages in the queue. Here is an example:

   while True:
       messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
       if 'Messages' in messages: # when the queue is exhausted, the response dict contains no 'Messages' key
           for message in messages['Messages']: # 'Messages' is a list
               # process the messages
               print(message['Body'])
               sqs.delete_message(QueueUrl=queue_url,ReceiptHandle=message['ReceiptHandle'])
       else:
           print('Queue is now empty')
           break
   

In this example, receive_message is used to get the messages from the queue, and delete_message is used to delete a message from the queue after it has been processed. The MaxNumberOfMessages parameter is set to 10, which is the maximum number of messages that can be retrieved in a single call.

Remember that you need to replace 'YOUR_QUEUE_NAME' with the name of your actual SQS queue.

This will help in processing and removing the messages from the queue thereby reducing the number of unprocessed messages.

Note: The above script assumes that you have appropriate permissions to read and delete messages from the queue. If you do not have these permissions, you will need to update your IAM policy accordingly.

Using Terraform

This finding cannot be remediated via Terraform, because Terraform only manages configuration of the SQS queue, not the runtime depth or rate of message processing.

A “high number of unprocessed messages” is an operational/runtime issue (consumer capacity, code errors, scaling, dead-letter queue configuration), not a queue property that the aws_sqs_queue resource can set. There is no Terraform argument to cap ApproximateNumberOfMessages or auto-drain/backpressure the queue.

To fix it you must change your application/consumer side and/or infrastructure (for example: increase consumer concurrency, autoscale workers based on CloudWatch metrics, fix processing errors, adjust visibility timeout, or route to a DLQ). Those changes are outside of Terraform’s ability to directly “drain” messages from an existing SQS queue.

Additional Reading:

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-sqs.html