0

I made MSK cluster, and sent a message from my Producer. It works well. And, I want to make my Consumer get the message. The Consumer in a python file, and the python file is in EC2. How can I run the python file and make my Consumer get the message? Actually I can run the file manually, but I want it run automatically after the Producer sends message.

LYG
  • 41
  • 1
  • 7
  • 1
    what's your Consumer doing - why does it need to run automatically? Are you trying to do stream processing on the data that's being produced? If you can edit your question to elaborate a bit more it will be easier to give a good answer. – Robin Moffatt Mar 02 '22 at 10:50

2 Answers2

0

Do you initiate a consumer in the Python file? Or the Python file is the business logic you want to execute on each Kafka message? It is not clear from your question.

If it's the former you basically need to run the script when the EC2 is bootstrapping. You can do that using user-data which is a bash script running when the instance initiated.

If it's the latter, you should initiate a consumer and pass the handler from the Python script to the consumer poll loop.

Noam Levy
  • 179
  • 7
0

Since you are running MSK, you can use a Python Lambda action without writing any Kafka consumer code or running an EC2 machine.

https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html

If you must use EC2, then you should run your consumer indefinitely, and have it wait for any message, not be dependent upon any specific producer action.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245