SQS with AWS Event Bridge

Question

I am trying to set up a demo environment to try out SQS as an AWS Event Bridge Source. I tried uploading few documents to SQS to see if Event Bridge detects any change, but I don't see any events triggered. How can I test SQS as a source with AWS Event Bridge?

Resources:
  Queue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub ${AWS::StackName}

  LambdaHandlerExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  EventConsumerFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaHandlerExecutionRole.Arn
      Code:
        ZipFile: |
          import json

          def lambda_handler(event, context):
              print("Received event: " + json.dumps(event, indent=2))

      Runtime: python3.7
      Timeout: 50

  EventRule:
    Type: AWS::Events::Rule
    Properties:
      Description: eventEventRule
      State: ENABLED
      EventPattern:
        source:
          - aws.sqs
        resources:
          - !GetAtt Queue.Arn
      Targets:
        - Arn: !GetAtt EventConsumerFunction.Arn
          Id: EventConsumerFunctionTarget

  PermissionForEventsToInvokeLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref EventConsumerFunction
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt EventRule.Arn

score 5 · Accepted Answer · answered Mar 08 '21 at 22:53

5

SQS data events (publishing new message) are not source events for Event Bridge (EB). Only management events can be picked up by EB, e.g.:

purging of the queue
creating of new queue
deletion of a queue

Also your event rule should be more generic for that:

  EventRule:
    Type: AWS::Events::Rule
    Properties:
      Description: eventEventRule
      State: ENABLED
      EventPattern:
        source:
          - aws.sqs
        # resources:
        #   - !GetAtt Queue.Arn
      Targets:
        - Arn: !GetAtt EventConsumerFunction.Arn
          Id: EventConsumerFunctionTarget

You can also enable CloudWatch trial and detect API events for the SQS. This should enable fetching more events.

answered Mar 08 '21 at 22:53

Marcin

215,873
14
235
294

Thank you for clarifying. Looks like it won't meet my requirement. – thotam Mar 08 '21 at 23:45
1

@thotam What would you try to achieve? If you want to publish messages to EB, you have to do: SQS -> Lambda -> EB. – Marcin Mar 08 '21 at 23:53
I am trying to see if I can leverage the Event bridge to scale Fargate containers based on the SQS Number of messages. Currently, I have the following flow: SQS --> Cloudwatch Alarm(ApproximateNumberOfMessagesVisible) --> SNS --> Lambda --> Fargate scaling. With Event Bridge, trying to implement this: SQS --> EventBridge --> Fargate – thotam Mar 09 '21 at 00:01
@thotam Sadly you can't do SQS->EB->Fargate. Why not do it directly, SQS->Fargate? – Marcin Mar 09 '21 at 00:06
We need short-lived Fargate containers to run only when there are messages in SQS. Lambda won't fit for our use case where our jobs run more than 15 minutes. – thotam Mar 09 '21 at 01:40
@thotam I mean you would have to use SQS->Lambda, and the lambda would publish msgs to your EB event bus. – Marcin Mar 09 '21 at 02:14
As Marcin said you can just use lambda as a proxy to do what you want. Just trigger a lambda when a message comes into sqs. Also @Marcin where is the documentation on the limits of sqs and event bridge it took me a while to find your answer. – Presley Cobb Aug 07 '21 at 23:40

Potato · Answer 2 · 2021-09-07T09:22:07.957

I might be late but this can benefit someone else, have a look at this: https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ecs-patterns.QueueProcessingFargateService.html

This will handle scaling of the Fargate container based on a number of messages in the SQS Queue.

a simplest stack can be defined using AWS CDK as following:

queue = sqs.Queue(stack, "Queue")

cluster = aws_ecs.Cluster(
            stack, 'FargateCluster'
        )

queue_processing_fargate_service = QueueProcessingFargateService(stack, "Service",
        cluster=cluster,
        memory_limit_mi_b=512,
        image=ecs.ContainerImage.from_registry("test"),
        command=["-c", "4", "amazon.com"],
        enable_logging=False,
        desired_task_count=2,
        environment={
            "TEST_ENVIRONMENT_VARIABLE1": "test environment variable 1 value",
            "TEST_ENVIRONMENT_VARIABLE2": "test environment variable 2 value"
        },
        queue=queue,
        max_scaling_capacity=5,
        container_name="test"
    )

Please add further details to expand on your answer, such as working code or documentation citations. — Community, Sep 05 '21 at 23:43
Hey I am trying to use this `QueueProcessingFargateService` but how the heck do I know what data from the queue makes it into my python function or for that matter how to get access to the message? The documentation around this thing is pretty scant — Will.Evo, Oct 06 '22 at 21:56

SQS with AWS Event Bridge

2 Answers2

Linked

Related