0

I am building a dead letter queue. What I am interested in implementing is the ability to forward a message in its entirety to another queue (when there is, for example, an error). In other words, I am interested in preserving the metadata of the message as well as its contents.

Here is some code to highlight the issue:

from confluent_kafka import Producer
producer = Producer({'bootstrap.servers': "host1:9092",'client.id': '0', 'auto.offset.reset': 'smallest'})
while True:
  msg = consumer.poll(timeout=1)
  if msg is None:
    continue
  else:
    producer.produce(topic='test_topic', value=msg )

However, I am unable to send this message. Converting the message to json returns the following when producing a message: TypeError: Object of type Message is not JSON serializable. If I try serializing to JSON - then it seems I am not able to get the full representation of the message using json.

I am using confluent kafka pytrhon. This is the consumer - https://github.com/confluentinc/confluent-kafka-python/blob/a5663da7ea76e58d02b13e4e6703ea6a9c52ec11/src/confluent_kafka/src/Consumer.c. Producer - https://github.com/confluentinc/confluent-kafka-python/blob/a5663da7ea76e58d02b13e4e6703ea6a9c52ec11/src/confluent_kafka/src/Producer.c

How would I be able to forward a message from one queue to another while including the message metadata?

Zeruno
  • 1,391
  • 2
  • 20
  • 39
  • Please share the producer implementation since this is where the error is. Did you keep something like `value_serializer=json.dumps` on your Producer? You don't need this if sending only bytes – OneCricketeer Nov 17 '21 at 21:46
  • @OneCricketeer Implementation specified. Actually to clarify, I am unable to send a message directly as cimpl.Message type. I am using json.dumps myself as an alternative but both are not working. – Zeruno Nov 18 '21 at 20:11
  • 1) What exactly do you mean "full representation of the message"? 2) If you want to send the value only, you'd need `value=msg.value` 3) If you want to copy the keys and headers and timestamp, you'll need more send parameters for those 4) If there's no processing in your Python code, you should really just use MirrorMaker for this – OneCricketeer Nov 19 '21 at 14:23
  • I want to send the message object directly without additional send parameters. If I could serialize the message with some inbuilt method, that would be great, but you can clearly see that that is not working (e.g. json.dumps(message) fails). – Zeruno Nov 20 '21 at 23:05
  • You cannot serialize the message class, as the error says. You need to pass the additional parameters. What's wrong with that? – OneCricketeer Nov 21 '21 at 07:50
  • Why would I not be able to serialize the class? As you stated earlier I "don't need this if sending only bytes". The benefits of not having write a custom serializer, when not necessary, are clear. – Zeruno Nov 21 '21 at 18:26
  • I'm saying you wouldn't need `json.dumps`, but you do still need `key=msg.key,value=msg.value`. You cannot send the entire Message class as just the value – OneCricketeer Nov 21 '21 at 18:43

0 Answers0