0

I have an application that uses AWS SQS with Lambda to process the messages pushed on the Queue. The Lambda keeps on polling the Queue, and when a new message appears it process the message.

For this scenario, is it possible to replace the SQS with Kafka on the AWS. In other words, can we use Kafka as a Queue for this use case?

gtango
  • 21
  • 4

1 Answers1

1

You absolutely can. Have a look at AWS Amazon Managed Streaming for Apache Kafka (Amazon MSK) . It's a managed service for Apache Kafka.

As for lambda triggers, unfortunately it's not a built in trigger. You can easily replicate the behaviour by using a periodically triggered lambda function that checks if the messsages are visible and then invokes the function that will process the message or processes the message directly. For some direction you can refer this official guide which sets up a similar pipeline, but for AWS MQ.

Mayank Raj
  • 1,574
  • 11
  • 13
  • It used to not be a trigger and we used to have to poll periodically. Someone in the last couple of years they changed that. Now my queue triggers my lambda, I've been running this in a production system for a few years now. – Rohit Chatterjee Jul 16 '20 at 12:56
  • 1
    @RohitChatterjee, behind the scenes it is still lambda polling the queue periodically i.e it is a pull-based architecture vs a push-based. Once the lambda service gets a message in the SQS Queue, it wraps it up in an event and triggers the function with it. – Mayank Raj Jul 16 '20 at 13:42
  • @gtango, if the answer did indeed address your question, do you mind accepting it. That way, others who stumble upon this can easily find the solution. Cheers. – Mayank Raj Jul 16 '20 at 17:29