0

I'm planning to build a critical component which produces the messages to kafka. I'm just thinking, is there any way that python multithreading will help us in writing the efficient kafka producer with asyncio, multithreading ?

Also here I'm planning to create a thread on demand, the use case is like kafka producer script needs to consume the textfiles and produce it kafka (catch here is, this needs to be done on demand). Design will be like, kafka producer script will read the ondemand request from redis/rabbitmq queue, once we got the request, I'm planning to create a thread for each request. request contain which textfile to read and send.

Is it possible to implement in python using multithreading and asyncio ? any help on this appreciated. thanks

Learnis
  • 526
  • 5
  • 25
  • There's an aiokafka python library, but Kafka isn't meant to be used for file transfer – OneCricketeer Feb 07 '21 at 23:24
  • I mean, there is another job which writes the result to text file, and python will tail and read that file and push it to kafka. – Learnis Feb 08 '21 at 08:36
  • I suggest using something like Filebeat for tailing files rather than rewrite the same in Python, but that's besides the point. Yes, Kafka producers are thread-safe – OneCricketeer Feb 08 '21 at 15:50

0 Answers0