1

I am trying to import SQS queues based on resource groups and add all the visible messages available among the queues. The problem is boto3 is not returning all the queues sometimes. I know it is the problem with boto3 because I am getting correct values most of the time when I run the same code. In my case, boto3 is supposed to return 3 queues but sometimes either be or two of the queues are missing. Here is the code I have written

import boto3
import time
from datetime import timedelta, datetime

#clients for SQS, Cloudwatch, AutoScalingGroups, and ResourceGroups
sqs = boto3.client('sqs')
cloudwatch = boto3.client('cloudwatch')
asg = boto3.client('autoscaling')
rg = boto3.client('resource-groups')

#Importing SQS Queues with the tag Env:Production
response = rg.list_group_resources(Group='env_prod')
resources = response.get('Resources')
    
    
soa = 0.0 
noi = 0.0 
ABPI = 100    

def SumOfAverages(resources, soa, response, cloudwatch):
    for idents in resources:
        identifier = idents.get('Identifier')
        resourcetype = identifier.get('ResourceType')
        if resourcetype == 'AWS::SQS::Queue':
            RArn = identifier.get('ResourceArn') 
            step0 = RArn.split(':')
            step1 = step0[5]
            response1 = cloudwatch.get_metric_statistics(
            Namespace='AWS/SQS',
            MetricName='ApproximateNumberOfMessagesVisible',
            Dimensions=[
                {
                    'Name': 'QueueName',
                    'Value': step1
                },
            ],
            StartTime=datetime.utcnow() - timedelta(minutes=1),
            EndTime=datetime.utcnow(),
            Period=60,
            Statistics=[
                'Average',
            ],
            
            Unit='Count'
            )
            datapoints = response1['Datapoints']
            for values in datapoints:
                averages = values['Average']
                soa += averages
               
    return(soa)

result = SumOfAverages(resources, soa, response, cloudwatch)
print(result)


Response:
[{'Timestamp': datetime.datetime(2021, 5, 8, 6, 34, tzinfo=tzutc()), 'Average': 122.0, 'Unit': 'Count'}]
[{'Timestamp': datetime.datetime(2021, 5, 8, 6, 34, tzinfo=tzutc()), 'Average': 101.0, 'Unit': 'Count'}]
[{'Timestamp': datetime.datetime(2021, 5, 8, 6, 34, tzinfo=tzutc()), 'Average': 94.0, 'Unit': 'Count'}]


[{'Timestamp': datetime.datetime(2021, 5, 8, 6, 34, tzinfo=tzutc()), 'Average': 122.0, 'Unit': 'Count'}]
[{'Timestamp': datetime.datetime(2021, 5, 8, 6, 34, tzinfo=tzutc()), 'Average': 101.0, 'Unit': 'Count'}]
[]
DC_Valluru
  • 83
  • 8
  • Check `NextToken` in the response to your `list_group_resources` request. – jarmod May 07 '21 at 15:25
  • @jarmod there is no attribute as "NextToken" in the response to list_group_resources – DC_Valluru May 07 '21 at 16:21
  • OK was just checking in case results were paginated. Do all of the SQS queues show up in the AWS console for resource groups? – jarmod May 07 '21 at 17:10
  • @jarmod yes all are them are displayed in the console and most of the time in boto3 too. The problem is sometimes some of the queues are returning null sets. – DC_Valluru May 08 '21 at 05:36
  • Now I have uploaded the responses to the same code. You can see that in the second response one of the queues returned a null set. – DC_Valluru May 08 '21 at 06:38

0 Answers0