How to pull logs out of s3 bucket using SQS

Question

I have an s3 bucket saving logs constantly on json.gz format, I´m trying to pull out those logs to an elastic agent host on a GCP cluster using SQS.

My configuration for the SQS is a default one, also I´m using the FIFO type. The Access policy configuration I used:

{
    "Version": "2012-10-17",
    "Id": "example-ID",
    "Statement": [
        {
            "Sid": "example-statement-ID",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "SQS-queue-ARN",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
                },
                "StringEquals": {
                    "aws:SourceAccount": "bucket-owner-account-id"
                }
            }
        }
    ]
}

The elastic agent is using the sqs url as reference to pull out those logs from the s3 bucket. I don´t know why the sqs isn´t working. Also It´s my first time using aws so I´m really lost.

What documentation are you following here? – jarmod Nov 11 '22 at 15:11 — jarmod, Nov 11 '22 at 15:11

score 0 · Accepted Answer · answered Nov 11 '22 at 15:34

SQS can't pull files from S3. You will need to set up notifications on your S3 bucket to publish an event to your SQS queue when a log is added (so presumably you will need s3:ObjectCreated notifications).

Note however that this notification doesn't include the actual file contents. It only has the details of the file so you'll need some other mechanism to then actually read the file and process it, such as a lambda function. You can find an example event here.

How to pull logs out of s3 bucket using SQS

1 Answers1