Kafka Python sending data with custom timestamp

Asked Jul 04 '18 at 18:11

Active Jul 04 '18 at 19:14

Viewed 1,695 times

I have JSON data with timestamp in it like this

a = {'key1':'value','timestamp':'123344453'}

I am using kafka python version 1.3.4.1

My producer code:

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8'))
for a in list_of_dicts:
    producer.send('topic', a, timestamp_ms=int(a['timestamp']))

My consumer code:

consumer = KafkaConsumer('topic')
for msg in consumer:

    mj = (msg.value.decode('utf8'))
    data = json.loads(mj)
    print(data)
    print(msg.timestamp)

I want the data to be sent as per the timestamp. But the data gets sent all at once to the consumer and consumer prints all the dictionaries from the list all at once. My data has timestamp with gap of 120 seconds. So if my timestamp for eg is 120 then the next timestamp would be 240. I need the consumer to consume in that manner and not all at once or the producer to send according any one of the two.

Also i am not able to understand the use of timestamp_ms from the docs. Just beginning with kafka so i have little idea about it. Pls help.