Get event/message on Kafka when new file on S3

Question

Im quite new to AWS and also new to Kafka (using Confluent platform and .NET) .

We will receive large files (~1-40+Mb) to our S3-bucket and the consuming side of this should process these files. We will have all our messaging over Kafka.

Ive read that you should not send large files over Kafka, but maybe Im misinformed here?

If we instead want to just get an event that a new file has arrived on our S3-bucket (and of course some kind of reference to it), how would we go about?

Answer 1

You can receive notifications about events that happen in your S3 bucket like when a new object is created/deleted etc.

From the S3 documentation (as of writing this), the following destinations are supported:

1. Simple Notification Service (SNS)
2. Simple Queue Service (SQS)
3. AWS Lamdba function

For instance, you can choose SQS as your S3 notification destination and use Kafka SQS Source Connector to stream the events to Kafka. Then you can write your Kafka consumer applications that react to this events.

And yes, it is not recommended to send large files over Kafka. Just send pointers to them and let the consumer application fetch the information using those pointers. If you are consumer wants to fetch some s3 objects, configure your consumer to use the S3 SDKs.

Useful resources:

1. Enabling event notifications in S3
2. S3 Notification Event Structure (JSON) with examples
3. Kafka SQS Source Connector