28

The documentation for AWS SNS and SQS have sections about message attributes. But there is no explanation how to have SQS message attributes when that queue is subscribed to a SNS topic.

Is there a way to configure AWS SNS to add particular message attributes to the SQS messages send via the subscription?

Arafat Nalkhande
  • 11,078
  • 9
  • 39
  • 63
Evgeny
  • 6,533
  • 5
  • 58
  • 64
  • When an Amazon SQS queue is subscribed to an Amazon SNS topic, the SQS message will contain the message sent to (and forwarded from) SNS. What particular of attributes did you want? – John Rotenstein May 29 '17 at 12:15
  • "SQS Message Attributes" is a thing. Separate from the message body. – Evgeny May 29 '17 at 12:24
  • That is correct. However, when SNS sends a message to SQS, it populates the SQS message body. What attributes from SNS were you hoping to have populated as an SQS message attribute? – John Rotenstein May 29 '17 at 12:25
  • 6
    I want to have certain pre-defined attributes on the SQS messages generated from this SNS topic. For example messages on one topic would add an attribute `path: foo`, another SNS topic sending to the same SQS queue would add an attribute `path: bar`. According to http://docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html there seems to be a reference, but nothing else is documented. `*Important* To use message attributes with Amazon SQS endpoints, you must set the subscription attribute, Raw Message Delivery, to True.` – Evgeny May 29 '17 at 12:28

4 Answers4

49

From aws documentation:

To use message attributes with Amazon SQS endpoints, you must set the subscription attribute, Raw Message Delivery, to True. For more information about raw message delivery, see Appendix: Large Payload and Raw Message Delivery. https://docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html https://docs.aws.amazon.com/sns/latest/dg/large-payload-raw-message.html

Added an example from real-life project. Hopefully it helps to clarify things. The message published to the sns topic is following:

aws sns publish --topic-arn arn:aws:sns:us-west-2:xxx:pollution-event --message '{"operatorId":3375001,"eventTypeId":1,"eventLevelId":1,"validFrom":"2018-03-10T09:00:00Z","validTo":"2018-03-11T09:00:00Z"}'  --message-attributes '{"Type" : { "DataType":"String", "StringValue":"Orchestration.Services.Model.Pollution.PollutionMessage"}}'

Enable Raw Delivery is false (default). The message received by sqs contains only content, there are no attributes

{
  "Type": "Notification",
  "MessageId": "78d5bc6f-142c-5060-a75c-ef29b774ec66",
  "TopicArn": "arn:aws:sns:eu-west-2:xxx:pollution-event",
  "Message": "{\"validFrom\": \"2018-03-10T09:00:00Z\",\"validTo\": \"2018-03-11T09:00:00Z\",\"eventLevelId\": 1,\"eventTypeId\": 1,\"operatorId\": 3375001}",
  "Timestamp": "2018-04-17T11:33:44.770Z",
  "SignatureVersion": "1",
  "Signature": "xxx==",
  "SigningCertURL": "https://sns.eu-west-2.amazonaws.com/SimpleNotificationService-xxx.pem",
  "UnsubscribeURL": "https://sns.eu-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-2:xxx",
  "MessageAttributes": {
    "Type": {
      "Type": "String",
      "Value": "Orchestration.Services.Model.Pollution.PollutionMessage"
    },
    "AWS.SNS.MOBILE.MPNS.Type": {
      "Type": "String",
      "Value": "token"
    },
    "AWS.SNS.MOBILE.MPNS.NotificationClass": {
      "Type": "String",
      "Value": "realtime"
    },
    "AWS.SNS.MOBILE.WNS.Type": {
      "Type": "String",
      "Value": "wns/badge"
    }
  }
}

There are no message attributes, it is contained within the message itself

Enable Raw delivery is true. The message contains the message attributes and proper content Message has an attribute The attribute contains expected value

Borys Generalov
  • 2,255
  • 17
  • 19
  • Have you tested this? According to John's answer, he writes at the very beginning that message attributes are delivered in the body of the SQS message and not as SQS attributes. – Evgeny Apr 10 '18 at 12:17
  • 2
    Yes, we had exactly the same issue: we published the message with an attribute, but sns forwarded it without any attribute. The sqs contained a message with json content and which in turn contained the attribute. Enabling Raw Message Delivery fixed the issue, i.e. sns sends the message as-is and the attributes remain – Borys Generalov Apr 11 '18 at 20:46
  • Did you publish a message with an SNS attribute? Because that would mean that you still did not get any SQS attributes set on that message, you just received the published SNS raw data. Not the same thing. – Evgeny Apr 12 '18 at 04:57
  • Ah sorry for confusion, would you adjust the question to make it more clear? As far as I understood you were sending the sns message with attributes, but these attributes were not sent to the sqs which is default behavior. Enabling Raw Delivery will deliver the message together with its attributes to the subscribed sqs queue. However, it seems like this is not what you are looking for. – Borys Generalov Apr 17 '18 at 13:12
  • 4
    This should be the accepted answer. Once you change the raw setting from default, the message attributes show up correctly in SQS. – Michael Oryl Oct 15 '18 at 12:59
  • 1
    Very weird. Although the attributes appear on the AWS console, they are not received via the SDK. Why could that happen? – k.liakos Feb 20 '19 at 21:37
  • 4
    Meh, you have to pass the --message-attribute-names parameter in order to receive the message attributes... – k.liakos Feb 20 '19 at 21:53
  • 1
    This is the correct answer. Enabling the `Raw message delivery` will "pass" the message attributes from SNS to SQS as actual message attributes instead of being part of the body. – IMB Aug 19 '19 at 16:10
  • Thanks - I can confirm this still is the case in 2020. While I can't see the explicit reference to the quote in the docs anymore, I can confirm the attribute filtering does not work if raw message delivery is not set. AWS needs to make this explicit in the docs – Avner Oct 20 '20 at 22:35
  • Great answer. Exactly what I needed for the `raw` message delivery — I need to add `--message-attribute-names=All` to `aws sqs receive-message` call. – Dmitriy Popov Aug 20 '21 at 15:28
25

Note: See other answer for better response, using Raw Message Delivery

While the Using Amazon SNS Message Attributes documentation sending Amazon SNS message attributes to Amazon SQS, it appears that the attributes are sent in the body of the message rather than being attached as message attributes on the resulting Amazon SQS messages.

For example, I did the following:

  • Created an Amazon SNS topic
  • Created an Amazon SQS queue and subscribed it to the SNS topic
  • Published a message to SNS

I published via the AWS Command-Line Interface (CLI):

aws sns publish --topic-arn arn:aws:sns:ap-southeast-2:123456789012:foo --message msg --subject subj --message-attributes '{"somename" : { "DataType":"String", "StringValue":"somevalue"}}'

(I got syntax help from map datatype in aws cli)

The resulting message in SQS showed the attributes as part of the message:

{
  "Type" : "Notification",
  "MessageId" : "53e3adad-723a-5eae-a7b7-fc0468ec2d37",
  "TopicArn" : "arn:aws:sns:ap-southeast-2:123456789012:foo",
  "Subject" : "subj",
  "Message" : "msg",
  "Timestamp" : "2017-05-29T12:48:22.186Z",
  ...
  "MessageAttributes" : {
    "somename" : {"Type":"String","Value":"somevalue"}
  }
}

It would be much nicer if these attributes were attached to the SQS message as official SQS attributes. Alas, it seems this is not the case.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • I assume from your answer, that at the moment there is no way to make SNS actually create SQS attributes in any way. And most probably the only way to add SQS attributes is to use one of the SDKs or the API directly without a service like SNS. – Evgeny May 29 '17 at 13:05
  • 1
    @Evgeny that's correct. Unfortunately, the SNS/SQS integration does not handle Message Attributes in the intuitive way. You could also, for example, use SNS > Lambda > SQS, with custom code in the Lambda function to populate the SQS Message Attributes from the SNS Message Attributes. – Michael - sqlbot May 29 '17 at 13:25
  • 1
    @Michael-sqlbot Already working on that. Appears that CloudWatch Events SQS target also does not support SQS _message attributes_, so working on that as well :( – Evgeny May 29 '17 at 13:27
  • 2
    Did you forget to change the subscription attribute RawMessageDelivery, which is false by default? Example using aws cli: aws sns set-subscription-attributes --subscription-arn 'subscriptionarn' --attribute-name RawMessageDelivery --attribute-value true – Borys Generalov Apr 11 '18 at 20:53
  • 8
    This is not the correct answer. @BorysGeneralov got it right in his answer: https://stackoverflow.com/a/49753291/1480995 – Michael Oryl Oct 15 '18 at 13:01
  • Not sure about the way and the specifics, if you're looking for a quick solution, I had to use this object signature (`network` is related to my project's business rule): ` MessageAttributes: { network: { DataType: 'String', StringValue: messageBody.network, }, }, ` – A Mehmeto Apr 13 '22 at 21:28
3

If you are here because you have a SQS queue that is subscribed to an SNS topic, you checked that your subscription has set the Raw Message Delivery to True but you still cannot read an attribute on your SQS message:

Make sure that your SQS client is not filtering out message attributes.

The code below will only include myAttribute when receiving messages from the SQS queue:

SQS.receiveMessage({
  QueueUrl: queueUrl,
  VisibilityTimeout: 20,
  WaitTimeSeconds: 10,
  MessageAttributeNames: [
    "myAttribute"
  ],
},...

If you want to read the value of some attribute other than myAttribute you will have to specify it (white list) or replace "myAttribute" with "All" to include all SQS attributes.

SQS.receiveMessage({
  MessageAttributeNames: [
    "myAttribute", "myOtherAttribute"
  ],
},...

Reference: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

PålOliver
  • 2,502
  • 1
  • 23
  • 27
2

Enabled Raw message delivery type while adding SQS subscription for the topic inside SNS

Pravin Bansal
  • 4,315
  • 1
  • 28
  • 19