0

I want to send two or more images simultaneously from Kafka Producer. Is it possibile?

After created a KafkaProducer object kafka_producer and after got frames from two webcam and converted them into bytes using cv2.imencode and tobytes(), I've tried this:

buffer_frame1 = cv2.imencode('jpg', frame1)
buffer_frame2 = cv2.imencode('jpg', frame2)

kafka_producer.send(topic, buffer_frame1.tobytes())
kafka_producer.send(topic, buffer_frame2.tobytes())

I haven't yet implement a KafkaConsumer, but I want to know if this work, and if it don't why and how can I send multiple images simultaneously.

Thanks!

jumpy
  • 73
  • 7

1 Answers1

0

You can encode both bytes objects into another object and then use a serialization library such as pickle (if you must stay within python) to send a single object over the wire.

If you must support other types of languages, JSON is perfectly capable of storing bytes

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • How do I encode both bytes objects into another object? – jumpy Mar 17 '20 at 09:31
  • You make another object that holds a list of images/bytes. Start here https://docs.python.org/3/tutorial/classes.html – OneCricketeer Mar 17 '20 at 10:38
  • Do I create an object with two attributes containing images that I will send? And then how can I serialize and de-serialize using Pickle? Can you give me an example? Now I'm sending images using two ```send``` in Producer and in Consumer I read them using ```for msg in self.consumer``` and then ```result = msg.value``` but how can I distinguish them and overwrite them when other two images are sended? – jumpy Mar 17 '20 at 13:23
  • Yup. And `pickle.dumps`, https://docs.python.org/3/library/pickle.html?highlight=pickle#module-pickle although, like I said, it's just one (uncommon) option. – OneCricketeer Mar 17 '20 at 13:26
  • Okay, I created an object with 2 attributes containing images bytes and then I used ```pickle.dumps``` to create bytes object and send it using ```send```. In the Consumer, when I try to read using ```for message in self.consumer``` it doesn't enter in the for loop. It seems that no message is published. Why this? – jumpy Mar 17 '20 at 13:44
  • Make sure you call producer.flush – OneCricketeer Mar 18 '20 at 07:00
  • It doesn't work. I put a ```print``` after ```for msg in consumer``` but it doesn't print anything – jumpy Mar 18 '20 at 07:41
  • Maybe the problem may be the size of images. I have two images, after encoded them the shapes are: (76270, 1) for the img1 and (921853, 1) for the img2. When I send only img1 the consumer received the message, but when I send also img2 the consumer doesn't receive anything. Another doubt is that before images are encoded, they have the same shape, but after encoded them their shapes are different, why this? Could be this the problem? – jumpy Mar 18 '20 at 10:38
  • Kafka has a default message size of 1MB. The broker will have logs saying that it is rejecting messages if you don't change those settings https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb I would need to see your code to know why shapes would change – OneCricketeer Mar 19 '20 at 00:10