0

I'm trying to figure out how to use the new destination feature, introduced here, to transfer data from one SQS queue, to another one, through a Golang lambda.

Set up

Following AWS documentation, I have build the following SAM template :

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  helloworld

Resources:
  helloworldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Timeout: 30
      CodeUri: helloworld/
      Handler: helloworld
      Runtime: go1.x
      Events:
        QueueEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt Tohelloworld.Arn
      Policies:
      - SQSSendMessagePolicy:
          QueueName:
            !GetAtt Fromhelloworld.QueueName

  EventInvokeConfig:
    Type: AWS::Lambda::EventInvokeConfig
    Properties:
      FunctionName: !Ref helloworldFunction
      Qualifier: "$LATEST"
      MaximumEventAgeInSeconds: 60
      MaximumRetryAttempts: 0
      DestinationConfig:
        OnSuccess:
          Destination: !GetAtt Fromhelloworld.Arn

  Tohelloworld:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 30
  Fromhelloworld:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 30

I have a trivial golang script, that always succeed :

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func handler(ctx context.Context, name events.SQSEvent) (int, error) {
    return 1, nil
}
func main() {
    lambda.Start(handler)
}

Everything is deployed through AWS SAM, ressources are reported to having been correctly created.

Problem

For each event put in the ToHelloWorldQueue I was expecting to get a message with 1 as body in the FromHelloWorldQueue.

Through SQS, I manually put an event in the ToHelloworld Queue, but the message never reaches the FromHelloworldQueue, even if the message has been correctly processed (as reported by CloudWatch). I even tried to manually set the policy to SQS:* for this lambda, but the problem still remains.

Can anyone help figuring out why I'm doing wrong please?

Edit:

Edit 1:

When directly invoked, the lambda function actually sends the message to the FromHelloWorld queue. So this just does not works when the lambda is triggered by the SQS ToHelloWorld queue.

jugo
  • 161
  • 2
  • 10
  • Trying to replicate your findings, I used your code and also built something Python based within the console. With none of those setups I were able to verify the destination working, neither when invoking via SQS nor triggering the lambda manually. How did you verify that the SQS queue received a new message? – Dennis Stritzke Dec 08 '19 at 07:12
  • I tried using ```aws sam invoke``` and directly by deploying the code. I finally figured out why this was not working. As stated in the documentation, the destination works when the Lambda is called in a asynchronously way. The ```aws sam invoke``` make an asynchroneous call IF called with an event payload. Otherwise, it's a synchroenous one. When triggered with SQS, the call is synchroneous, hence, the request's result was never forward to the destination queue. – jugo Dec 09 '19 at 00:44
  • That is interesting. Could you post the solution you found as the answer to your question? I would particularly be interested why a trigger through SQS is synchronous. I thought we use SQS because it enables asynchronous processing... – Dennis Stritzke Dec 09 '19 at 06:09
  • Actually, I don't have any solution yet : as a workaround, I'm manually sending the event to the ```FromHelloWorld``` queue through the sdk. Regarding the Sync/Async stuff (and from my, imperfect, understanding) : since SQS is a Queue, events will still be processing asynchroneously, from the sender perspective, but that does not imply that the lambda is necessarily called asynchroneously, from the queue perspective. – jugo Dec 09 '19 at 15:07
  • Hmm, it's unfortunate that you have to replicate what this feature should be for. Your explanation about sync/async sounds plausible. Let's see what some experimentation and time will show... – Dennis Stritzke Dec 10 '19 at 20:17

0 Answers0