0

I wish to consume a specific number of maximum messages per consume from a kafka topic. I have used the following code:

consumer = KafkaConsumer(topic, bootstrap_servers=[server], max_poll_records=2, api_version=(0, 10))

for message in consumer:
    string = message.value.decode("utf-8")
    dict_value = ast.literal_eval(string)

But by doing this I am not able to restrict the maximum messages per consume to two.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Suganth
  • 35
  • 1
  • 7

1 Answers1

1

This waits to poll two records each fetch request,and will continue indefinitely until the application is killed.

As answered before, collect events to a list + break the loop after you get only two messages.

Or use something like this

consumer = KafkaConsumer(...)
m1 = next(consumer)
m2 = next(consumer)

Or modify the producer code to produce two events together so that the consumer reads them as "one" record

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245