1

I have Flask application running to serve the API endpoint, in the same application I have to implement also kafka consumer in order to consume events from kafka stream, but because of the kafka consumer should run forever, I am using python threads to run the kafka consumer in the background, I have some questions please

  • is using threading ok? and What is the best practice to run Kafka consumer with flask app in general ?
  • If using threading is ok, how to force the flask app to stop if the kafka consumer thread stops for any reason ?
kichik
  • 33,220
  • 7
  • 94
  • 114
MuGh
  • 45
  • 3

1 Answers1

0

While possible, I would recommend two separate processes, rather than embed the Kafka consumer in the Flask app. You could use supervisor to run Kafka and Flask/gunicorn processes both at the same time.

Otherwise, for example, use Kafka Connect to write to a database. Have Flask query the database. Or externally run ksqlDB or Kafka Streams Interactive Query feature (requires JVM development, not Python) and query that.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245