python KafkaConsumer and Producer at the same time

Question

In a python program I would like to write some messages to Kafka, then read the response with the same number of messages from a remote app on a different topic. The problem is that by the time I am done with sending messages, the other end already start responding and when I start reading, I only get the tailing part of the message batch, or no messages at all, depending on timing. This contradict to my understanding of the package, i.e. I thought if I create a consumer with auto_offset_reset='latest', and subscribe for a topic, then it remembers the offset at subscription time and when I get to iterate over the consumer object, it will start reading messages from that offset.

Here is what I do:

I create a consumer first and subscribe for the out topic:

consumer = KafkaConsumer(
  bootstrap_servers=host+':'+broker_port, 
  group_id = "0", 
  auto_offset_reset='latest',
  consumer_timeout_ms=10000
)
consumer.subscribe(topics=(topic_out))

then create a producer and send messages to topic_in:

producer = KafkaProducer(
   bootstrap_servers=host+':'+broker_port
)
future = producer.send(topic,json.dumps(record).encode('utf-8'))
future.get(timeout=5)

Then I start reading from the consumer:

results = []
for msg in consumer:
   message = json.loads(msg.value)
   results.append(message)

I tried consumer.pause() before sending, consumer.resume() after sending - does not help.

Is there something I am missing in the configuration, or I misunderstand how Consumer works?

Thanks in advance!

is your consumer is up and running before producer sends the data? and you are saying consumer is not consuming the messages if i understand it correctly — Ryuzaki L, Sep 16 '19 at 14:40
Sorry if not clear: C and P are both in the same programm:. I first instantiate a C, then P, then send messages with P, and then reading messages with previously instantiated C — Artem Trunov, Sep 16 '19 at 14:44
Then change this value `group_id = "0"` and then restart the process — Ryuzaki L, Sep 16 '19 at 14:45

score 0 · Answer 1 · answered Dec 31 '19 at 14:51

Sounds like you have a race condition.

One solution would be to store a local dictionary or sqlite table that's built from consuming this "lookup topic", then when you consume from the "action topic", you are doing lookups locally rather than starting a consumer to scan over the "lookup topic" for the data you need.

python KafkaConsumer and Producer at the same time

1 Answers1