0

Defaults must be bad or something is very wrong.

I run produce and consumer in separate REPL.

import datetime
import uuid
import time
import json

import msgpack

from kafka import KafkaProducer, KafkaConsumer
from kafka.errors import KafkaError

topic = 'my-topic'
def get_consumer():
    return KafkaConsumer(topic,
            # client_id=str(uuid.uuid1()),
            fetch_max_wait_ms=1,
            group_id='my-group',
            metadata_max_age_ms=10,
            bootstrap_servers=['localhost:9092'],
            value_deserializer=msgpack.loads)

consumer = get_consumer()

producer = KafkaProducer(
        bootstrap_servers=["localhost:9092"],
        value_serializer=msgpack.dumps)

def produce_n(n=100):
    for i in range(n):
        print(i)
        producer.send(topic, value=time.time())
        time.sleep(1)
        producer.flush()

def consume():
    consumer = get_consumer()
    for x in consumer:
        T = time.time()
        dt = T - x.value
        print(x)
        print(f't={x.value} T={T} dt={dt} ms')

mathtick
  • 6,487
  • 13
  • 56
  • 101
  • can u pls add more information , as to what is happening? Your question lacks any details to help. – Indraneel Bende Jan 04 '20 at 05:31
  • The consumer takes a very long time to start printing. There is nothing that is not in the script. You start "produce_n()" in one REPL and consume() in another. It does not start printing for a long time. Often I don't see anything for minutes and give up. The logs from kafka do not seem relevant but I could be wrong. – mathtick Jan 04 '20 at 11:19
  • Why is there a sleep in ur producer? Remove the flush too. – Indraneel Bende Jan 04 '20 at 16:38
  • So basically ur point is after u start producer and consumer together , it takes a very long time to see the print logs in the consumer , long after u have produced the records? – Indraneel Bende Jan 04 '20 at 16:39
  • @IndraneelBende yes that is it. And I can start and stop the producer and consumer etc but hasn't fixed it yet. – mathtick Jan 05 '20 at 14:16
  • Why is there a sleep in the producer? – Indraneel Bende Jan 05 '20 at 14:17
  • That is just to limit the amount of stuff I was pushing through. My goal was to check latency, which ends up being ok (10 microseconds) *once* data starts to come through. – mathtick Jan 05 '20 at 14:55
  • You don't need to sleep or flush . Could u remove the sleep and flush ? – Indraneel Bende Jan 05 '20 at 14:56
  • I wouldn't say that is related to the issue but there is no need to have it. There does not seem to be anything wrong in the code . Remove the sleep and flush , and try again pls. – Indraneel Bende Jan 05 '20 at 14:58
  • So I restarted the whole (single machine docker "cluster") and tried all the combinations (no flush, no sleep etc) and now it seems to always be reasonably working. Meaning I haven't been able to reproduce the problem in the last five minutes. Will post back if I figure it out. I suspect I there was some state in the system I had not reset. – mathtick Jan 05 '20 at 15:11
  • How many brokers u have in ur setup and what is ur in sync replica setting? – Indraneel Bende Jan 05 '20 at 15:13
  • The consumers don't consume a message unless it is 'committed' , that is it is replicated to all in sync replicas . Could this be a replication issue? – Indraneel Bende Jan 05 '20 at 15:14
  • I have a super simple setup with one kafka server and one zookeeper. Could be the commit issue but I see this (small) gap sometimes again now, particularly when I "reload" the python module and change the topic and try to produce/consume again: will post in next comment – mathtick Jan 05 '20 at 15:27
  • [2020-01-05 15:24:48,187] INFO [GroupCoordinator 0]: Preparing to rebalance group my-group in state PreparingRebalance with old generation 12 (__consumer_offsets-12) (reason: Adding new member kafka-python-1.4.6-7da8ac1b-d7dd-4ece-96e1-70a810389fa4 with group instanceid None) (kafka.coordinator.group.GroupCoordinator) [2020-01-05 15:24:54,703] INFO [GroupCoordinator 0]: Member kafka-python-1.4.6-9f1cc5bf-2060-416f-8b1d-4e7e1dada539 in group my-group has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) – mathtick Jan 05 '20 at 15:27
  • 1
    This seems to be a log indicating a rebalance is happening on the consumer group – Indraneel Bende Jan 05 '20 at 16:57

0 Answers0