28

I created a SNS topic that publishes all the information coming out of Cloudformation via the cli. However, when I check the queue, it is not receiving any of the SNS messages. I verified the SNS is working by subscribing my email to it, so the issue seems to be in the connection between the queue and the SNS. However, I cannot find any problems with my syntax. I, as far as I know, have followed amazon's documentation precisely.

Bash:

#SNS parameters
SNS_NAME="${NAME}_SNS"
SQS_NAME="${NAME}_SQS"

#Create SNS topic to send cloudformation notifications to
SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`

#Create SQS to send SNS to (holding SNS messages for lambda -^ up)
SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`

#subscribe the queue to the notifications
aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}
aws sns subscribe --topic-arn ${SNS_ARN} --protocol email-json --notification-endpoint ${EMAIL}

#Create the stack which kicks everything else off-
aws cloudformation create-stack $REGIONTEXT $ITYPETEXT --capabilities CAPABILITY_IAM --template-url https://${BUCKETNAME}.s3.amazonaws.com/${TEMPLATE} --notification-arns ${SNS_ARN} --stack-name $NAME --parameters ParameterKey=SNSARN,ParameterValue=${SNS_ARN} ParameterKey=Bucket,ParameterValue=${BUCKETNAME} ${PARAMTEXT} ${EXTRAARGS}
Mark B
  • 183,023
  • 24
  • 297
  • 295
asdf
  • 2,927
  • 2
  • 21
  • 42

4 Answers4

38

It doesn't look like you have given the SNS topic permission to publish to the SQS queue. Look at step 2 in this walkthrough. You'll need to add a policy like this to the SQS queue:

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"MySQSPolicy001",
      "Effect":"Allow",
      "Principal":"*",
      "Action":"sqs:SendMessage",
      "Resource":"arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "Condition":{
        "ArnEquals":{
          "aws:SourceArn":"arn:aws:sns:us-east-1:123456789012:MyTopic"
        }
      }
    }
  ]
}

Replacing the ARNs with the ones for your topic and queue.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Is there any way to do that via the CLI or will it have to be in a cloud formation template? – asdf Aug 04 '16 at 19:40
  • Yes of course you can use the CLI. You aren't forced to use Cloud Formation for anything. This is the CLI command you would need to look at http://docs.aws.amazon.com/cli/latest/reference/sqs/set-queue-attributes.html – Mark B Aug 04 '16 at 19:56
  • Solved it wonderfully. However, I am going to add my own answer to clear up issues with using bash + cli in this case as theirs a lot of funkiness not covered in the docs. – asdf Aug 04 '16 at 23:12
  • helped me find a typo in my terraform code... thanks. – weaveoftheride Jun 10 '20 at 15:09
  • I'm currently using similar sort of SQS policy and it is working fine. But I want have principal as accountID and to remove the condition part If i do so i'm not getting the messages. Is there any specific reason? – Prashanna Feb 12 '21 at 09:09
  • Yep, it was a permissions issue for me too. – CodeConnoisseur Aug 05 '22 at 02:28
18

In my case SQS wan't receiving messages from SNS because SQS had encryption turned ON. When I turned OFF encryption on SQS it started working!

This AWS documentation explains how to enable SNS compatibility with encrypted SQS queues:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html#compatibility-with-aws-services

SNS requires extra permissions to be able to use the KMS key to encrypt messages for the queue.

thenickdude
  • 1,640
  • 1
  • 17
  • 18
Krisso
  • 181
  • 1
  • 2
8

If the SQS is encryted then the event pushing messages to queue must follow below steps

Several AWS services send events to Amazon SQS queues. To allow these event sources to work with encrypted queues, you must perform the following steps.

  • Use a customer managed CMK.

    To allow the AWS service to have the kms:GenerateDataKey* and kms:Decrypt permissions, add the following statement to the CMK policy.

    { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "service.amazonaws.com" }, "Action": [ "kms:GenerateDataKey*", "kms:Decrypt" ], "Resource": "*" }] }

  • Create a new SSE queue or configure an existing SSE queue using the ARN of your CMK.

    Provide the ARN of the encrypted queue to the event source.

For sns replace service section with sns.amazonaws.com

"Principal": { "Service": "sns.amazonaws.com" }

kartick shaw
  • 915
  • 13
  • 5
6

Thank you to Mark B for his answer. It provided the start to getting this working. However, in order to make a policy document work via the CLI there are a few quirks that aren't covered in the docs.

  1. There are all sorts of errors trying to pass json directly to the --attributes flag in aws sqs set-queue-attributes command. For some reason it requires the modifying json to be in a .json document referenced by the cli.
  2. In the .json file provided to the cli, all of the double quotes inside the "Policy" value (nested json) must be escaped (i.e. { \"Statement\": \"HelloWorld\" }). If this is not followed, it will validation errors. I ended up needing to use the ascii escape characters in order properly format the output (\x5C).
  3. The json file must be referenced by using file://local-location in the --attributes flag. It throws errors if this is not followed.

See the following elements I used for reference:

load_sqs.sh:

SQS_POLICY=
sqs-policy()
{
#First param is the queue arn, second param is the topic arn
SQS_POLICY=`printf '{ "Policy": "{\x5C\"Version\x5C\":\x5C\"2012-10-17\x5C\",\x5C\"Statement\x5C\":[{\x5C\"Sid\x5C\":\x5C\"CloudformationLambdaSQSPolicy\x5C\",\x5C\"Effect\x5C\":\x5C\"Allow\x5C\",\x5C\"Principal\x5C\":\x5C\"*\x5C\",\x5C\"Action\x5C\":\x5C\"sqs:SendMessage\x5C\",\x5C\"Resource\x5C\":\x5C\"%s\x5C\",\x5C\"Condition\x5C\":{\x5C\"ArnEquals\x5C\":{\x5C\"aws:SourceArn\x5C\":\x5C\"%s\x5C\"}}}]}" }' "$1" "$2"`
`echo $SQS_POLICY > $PWD/sqs-policy.json`
}

#SNS parameters
SNS_NAME="${NAME}_SNS"
SQS_NAME="${NAME}_SQS"

#Create SNS topic to send cloudformation notifications to
SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`

#Create SQS to send SNS to (holding SNS messages for lambda -^ up)
SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`

#Add necessary SQS <--> SNS permissions
sqs-policy ${SQS_ARN} ${SNS_ARN}
`aws sqs set-queue-attributes --queue-url ${SQS_URL} --attributes file://sqs-policy.json`

#subscribe the queue to the notifications
aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}

sqs-policy.json:

{ "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"CloudformationLambdaSQSPolicy\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"ResourceARN\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"SourceARN\"}}}]}" }
asdf
  • 2,927
  • 2
  • 21
  • 42