AWS API-Gateway communicating to SNS

Question

I am building an API which will be serviced by Lambda functions but I need these to be asynchronous so rather than connecting the API-Gateway directly to the Lambda function I'm using the "AWS Service Proxy" to publish SNS messages and then have the Lambda function subscribe to the relevant SNS topic so it receives delivery of the requests. Here's a picture which illustrates the flow:

I have tested both the Lambda function in isolation as well pub/sub messaging between SNS and Lambda but I am struggling with the API-Gateway to SNS handoff. Documentation is quite light but what I am assuming right now is that the following attributes must be sent in the POST request:

1. Action: the API-Gateway offers to set this in the UI and I have put in the Publish action which is the appropriate SNS action

2. Message: the body of the POST message should be a JSON document. It would be passed by the web client and proxied through the gateway to SNS.

3. TopicArn: indicates the SNS topic that we're publishing to. In my design this would be a static value/endpoint so I'd prefer that the web-client not have to pass this too but if it were easier to do this that would be fine too.

I have tried lots of things but am just stuck. Would love to find a good code example somewhere but any help at all would be appreciated.

---

Wanted to add a little more context on my current attempt:

I have tried publishing my API and using Postman to try and get a valid response. Here's the postman screens(one for header vars, one for JSON body):

This results in the following error message:

{
   "Error": {
     "Code": "InvalidParameter",
     "Message": "Invalid parameter: TopicArn or TargetArn Reason: no value for required parameter",
     "Type": "Sender"
  },
  "RequestId": "b33b7700-e8a3-58f7-8ebe-39e4e62b02d0"
}

the error seems to indicate that the TopicArn parameter is not being sent to SNS but I have included the following in API-Gateway:

Answer 1

I'm from the Api Gateway team.

I believe there are a few formats for the HTTP request to the Publish API, but here's the one I used first:

AWS Region us-west-2

AWS Service sns

AWS Subdomain

HTTP method POST

Action Publish

== query strings ==

Subject 'foo'
Message 'bar'
TopicArn 'arn:aws:sns:us-west-2:xxxxxxxxxxxx:test-api'

This worked for me to publish a message.

Let me know if you have further troubles.

Jack

Answer 2

If anybody is still looking for a solution to the original problem, proxying a JSON request body to an SNS topic via API gateway alone, it's possible.

Create the gateway as Ken describes above. Then simply proxy the body to the Integration Request's query parameters. You can also hard code Subject, TopicArn, etc here, or map those from the request's body using a JsonPath.

For example:

{
   //body
   "topic": "arn:aws:sns:1234567:topic"
}

Could be mapped to a header as:

method.request.body.topic

Answer 3

I did eventually get this to work after working with AWS support. Here's my solution:

• First of all even though you're sending a POST you will not be able to send a JSON message in the body of the message as you might expect
• Instead you must URL Encode the JSON and pass it as query parameter
• Also remember that the JSON you send should start with a root object of default which in SNS-world means the "default channel"
• Then, eventually Lambda picks up the SNS event you must also abstract away a lot of the noise to get at your JSON message. For this I created the following function that I use within my Lambda function:

/**
 * When this is run in AWS it is run "through" a SNS
 * event wconfig.ich adds a lot of clutter to the event data,
 * this tests for SNS data and normalizes when necessary
 */
function abstractSNS(e) {
  if (e.Records) {
    return JSON.parse(decodeURIComponent(e.Records[0].Sns.Message)).default;
  } else {
    return e;
  }
}

/**
 * HANDLER
 * This is the entry point for the lambda function
 */
exports.handler = function handler(event, context) {
  parent.event = abstractSNS(event);

Answer 4

You could use API Gateway to invoke your Lambda function asynchronously by configuring it as an AWS service proxy. The configuration is basically the same you see in this GitHub sample, with the exception that the uri for the Lambda invocation changes to /invoke-async/ instead of just /invoke/

Answer 5

Here is a step by step guide for people who still couldn't figure it out after looking through the above answers. The variable names are case sensitive so make sure you are exact.

1. Open the Post Method
   a. Select Method Request
   b. Change Request Validator to Validate body, query string parameters, and headers
   c. Expand URL Query String Parameters
   d. Add the following two query string parameters
   Name: TopicArn ----> Select Required
   Name: Message -----> Select Required

2. Return to the Post Method and open Integration Request
   a. Expand URL Query String Parameters
   b. Add the following two query string parameters
   Name: TopicArn Mapped from: method.request.querystring.TopicArn
   Name: Message Mapped from: method.request.querystring.Message

3. When testing, alter the below command to match your SNS ARN and put it in Query Strings.
   TopicArn=arn:aws:sns:us-west-2:1234567890:SNSName&Message="Hello from API Gateway"

Sources/Further Information:
API Gateway Proxy Integration Service Guide
SNS Publish Method Documentation

Answer 6

I am just speculating (haven't tried this myself), but I think you are not sending the message correctly...

Based on AWS's documentation here (http://docs.aws.amazon.com/sns/latest/api/API_Publish.html), you need to POST the message in what seems to be the application/x-www-form-urlencoded encoding like this:

POST http://sns.us-west-2.amazonaws.com/ HTTP/1.1
...
Action=Publish
&Message=%7B%22default%22%3A%22This+is+the+default+Message%22%2C%22APNS_SANDBOX%22%3A%22%7B+%5C%22aps%5C%22+%3A+%7B+%5C%22alert%5C%22+%3A+%5C%22You+have+got+email.%5C%22%2C+%5C%22badge%5C%22+%3A+9%2C%5C%22sound%5C%22+%3A%5C%22default%5C%22%7D%7D%22%7D
&TargetArn=arn%3Aaws%3Asns%3Aus-west-2%3A803981987763%3Aendpoint%2FAPNS_SANDBOX%2Fpushapp%2F98e9ced9-f136-3893-9d60-776547eafebb
&SignatureMethod=HmacSHA256
&AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE
&SignatureVersion=2
&Version=2010-03-31
&Signature=vmqc4XRupKAxsDAdN4j4Ayw5LQljXMps3kss4bkDfCk%3D
&Timestamp=2013-07-18T22%3A44%3A09.452Z
&MessageStructure=json

That is, the message body looks the way a browser would encode form data. Your message can be JSON formatted, but still needs to be encoded as if it was a form field (an awkward analogy :)).

Also, based on the common parameters documentation (http://docs.aws.amazon.com/sns/latest/api/CommonParameters.html), you have a number of additional required fields (the usual access key, signature and so on).

You have not specified what language you are writing your API Gateway in - there might be an AWS SDK for it that you can use, instead of trying to manually compose the REST requests).

Answer 7

I would do it like:

WebApp --> Gateway --> Lambda ( Use Boto3 to publish in SNS ) --> SNS -->Lambda

I think, things will be simpler.

Answer 8

Also remember parameters are case sensitive; I also received the OP's error: "Message": "Invalid parameter: TopicArn or TargetArn Reason: no value for required parameter"

The only problem was case sensitivity of the parameters (specifically it should be: "TopicArn" and "Message"). These are set in the Method Execution | POST - Integration Request section, in the "Name" field.

The capitalization of the "Mapped from" is important that it matches the params being sent through from the Method Request configuration, but what is sent on to SNS is the "Name" field of "Integration Request", and that is what I had wrong.

Answer 9

If anyone's looking for an example of this in 2023 using Terraform with the OpenAPI configuration, here's what I used

resource "aws_api_gateway_rest_api" "MyApi" {
  name = "${var.name}-${var.environment}"
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : aws_iam_role.api-gateway-access-role.arn
        },
        "Action" : "execute-api:Invoke",
        "Resource" : "arn:aws:execute-api:${var.aws_region}:${var.aws_account_id}:*/*/*/*"
        "Condition" : {
          "StringEquals" : {
            "aws:SourceVpc" : data.terraform_remote_state.vpc.outputs.vpc_id
          }
        }
      }
    ]
  })
  body = jsonencode({
    "openapi" : "3.0.1",
    "info" : {
      "title" : "MyApi",
      "version" : "0.1.0"
    },
    "paths" : {
      "/my-in-principle" : {
        "post" : {
          "x-amazon-apigateway-auth" : {
            "type" : "AWS_IAM"
          },
          "x-amazon-apigateway-integration" : {
            "uri" : "arn:aws:apigateway:${var.aws_region}:sns:path/${aws_sns_topic.my-request.arn}?Action=Publish&TopicArn=${aws_sns_topic.my-request.arn}",
            "passthroughBehavior" : "when_no_templates",
            "credentials" : aws_iam_role.sns-access-role.arn,
            "httpMethod" : "POST",
            "type" : "aws",
            "requestTemplates" : {
              "application/json" : "{\"Message\": $input.json('$')}"
            },
            "requestParameters" : {
              "integration.request.querystring.Message" : "method.request.body"
            }
            "responses" : {
              "400" : {
                "statusCode" : "400",
                "contentHandling" : "CONVERT_TO_TEXT"
                "responseTemplates" : {
                  "application/json" : "$input.json('$')"
                },
              },
              "500" : {
                "statusCode" : "500",
                "contentHandling" : "CONVERT_TO_TEXT"
                "responseTemplates" : {
                  "application/json" : "$input.json('$')"
                },
              },
              "200" : {
                "statusCode" : "200",
                "contentHandling" : "CONVERT_TO_TEXT"
                "responseTemplates" : {
                  "application/json" : "$input.json('$')"
                },
              }
            }
          },
          "responses" : {
            "200" : {
              "description" : "Successful Response"
            },
            "400" : {
              "description" : "Bad Request"
            },
            "500" : {
              "description" : "Internal Server Error"
            }
          }
        }
      }
    }
  })
  endpoint_configuration {
    types            = ["PRIVATE"]
    vpc_endpoint_ids = [data.aws_vpc_endpoint.execute-api.id]
  }
}

resource "aws_sns_topic_policy" "my-request-topic-policy" {
  arn = aws_sns_topic.my-request.arn
  policy = jsonencode({
    Version = "2012-10-17",
    Id      = "default",
    Statement = [
      {
        Sid    = "AllowAPIGatewayToPublish",
        Effect = "Allow",
        Principal = {
          Service = "apigateway.amazonaws.com"
        },
        Action   = "sns:Publish",
        Resource = aws_sns_topic.my-request.arn,
        Condition = {
          ArnEquals = {
            "aws:SourceArn" = "arn:aws:execute-api:${var.aws_region}:${var.aws_account_id}:${aws_api_gateway_rest_api.MyApi.id}/*/*/*"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role" "sns-access-role" {
  name = "sns-access-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Service = "apigateway.amazonaws.com"
        },
        Action = "sts:AssumeRole"
      }
    ]
  })
  inline_policy {
    name = "sns-publish-policy"
    policy = jsonencode({
      Version = "2012-10-17",
      Statement = [
        {
          Effect = "Allow",
          Action = [
            "sns:Publish"
          ],
          Resource = [
            aws_sns_topic.my-request.arn
          ]
        }
      ]
    })
  }
}

resource "aws_api_gateway_method_settings" "MyMethodSettings" {
  rest_api_id = aws_api_gateway_rest_api.MyApi.id
  stage_name  = aws_api_gateway_stage.MyApiStage.stage_name
  method_path = "*/*"

  settings {
    throttling_rate_limit  = "5"
    throttling_burst_limit = "20"
  }
}

resource "aws_api_gateway_deployment" "MyDeployment" {
  rest_api_id = aws_api_gateway_rest_api.MyApi.id

  triggers = {
    redeployment = sha1(jsonencode([
      aws_api_gateway_rest_api.MyApi.body,
      aws_api_gateway_rest_api.MyApi.policy
    ]))
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_stage" "MyApiStage" {
  rest_api_id          = aws_api_gateway_rest_api.MyApi.id
  stage_name           = "v1"
  deployment_id        = aws_api_gateway_deployment.MyDeployment.id
  xray_tracing_enabled = true
}

resource "aws_lambda_permission" "MyLambdaInvokePermission" {
  statement_id  = "AllowExecutionFromAPIGateway"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.Async-Lambda.function_name
  principal     = "apigateway.amazonaws.com"

  # More: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-control-access-using-iam-policies-to-invoke-api.html
  source_arn = "arn:aws:execute-api:${var.aws_region}:${var.aws_account_id}:${aws_api_gateway_rest_api.MyApi.id}/*/*/*"
}