2

We have sluggish delivery times in a data pipeline that includes a SQS queue. So I want to know how much time the messages spend in the SQS bit.

CloudWatch only gives me not so useful gauges:

ApproximateNumberOfMessagesNotVisible
ApproximateNumberOfMessagesVisible
NumberOfEmptyReceives
NumberOfMessagesDeleted
NumberOfMessagesReceived
NumberOfMessagesSent
SentMessageSize

Currently there's no congestion, the queue is 99.99% of the time empty, but it's not a guarantee that each message spends a small time in the queue.

Is there any advanced metrics option or any external tool or Scala/Java library to measure Tconsume-Tproduce of each message ID?

sscarduzio
  • 5,938
  • 5
  • 42
  • 54

1 Answers1

1

There is no advance metric which will give you this information.

You will have to add logic to log these metrics when processing a message. There are additional attributes on an sqs message which can help you compute the time spent by the message on the queue before deletion.

CreatedTimestamp, LastModifiedTimestamp attributes on the message will allow you log the metric, you will then have to aggregate these metrics.

Shibashis
  • 8,023
  • 3
  • 27
  • 38
  • I only found references to those attributes referred to the queue, not to each message. http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html I'd need those timestamps for each message to compute the described metric. – sscarduzio Jul 29 '16 at 15:24
  • Wait, you were right somehow, I found my answer here: http://stackoverflow.com/questions/25107605/amazon-sqs-java-sdk-cannot-receive-message-attributes – sscarduzio Jul 29 '16 at 15:52
  • The java documentation for the API list all the attributes you can fetch.http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/ReceiveMessageRequest.html#withAttributeNames-com.amazonaws.services.sqs.model.QueueAttributeName...- – Shibashis Jul 29 '16 at 19:46