how to send Json file to kafka from Python

Question

I have a json file like below which I want to send to kafka from python.

Json File

 filename = 'External_Risk_{}.json'.format(date.today().strftime("%Y%m%d"))
    d =df.to_json(filename, orient='records')

Sending to Kafka

from kafka import SimpleProducer, KafkaClient
import json 
from kafka import KafkaProducer
producer =KafkaProducer(bootstrap_servers='xxx.xxx.xxx.xxx')
jd = json.dumps(d)
producer.send_messages(b'message1',jd)

But it not working . What is correct way of sending a json file to Kafka.

Why are you using pandas? If you have a dataframe, you should be sending each row separately, not the whole thing at once — OneCricketeer, Nov 22 '19 at 06:56
@cricket_007, the expected output inside kakfa is json and I am creating the json after many data manipulations using pandas. So should I be using your method to send the dataframe to kakfa. — Rahul rajan, Nov 22 '19 at 09:33

score 2 · Accepted Answer · answered Nov 22 '19 at 06:58

You should be asking how to load a file to a string, then you're just sending a string to Kafka

import json 
from kafka import KafkaProducer
producer =KafkaProducer(bootstrap_servers='xxx.xxx.xxx.xxx')

with open(filename) as f:
    data = json.load(f)
    producer.send_message(topic, data.encode('utf-8')

score 1 · Answer 2 · answered Nov 22 '19 at 06:28

1

Try this

from confluent_kafka import Producer
import json

p = Producer({'bootstrap.servers': 'localhost:9092'})

p.produce('topic', json.dumps({"demo": "message"}))

answered Nov 22 '19 at 06:28

sky

260
3
12

install confluent kafka using pip install confluent-kafka – sky Nov 22 '19 at 06:30
not its the topic on kafka on which you want to dump the json message. – sky Nov 22 '19 at 06:34
like from your example 'message1' is the topic – sky Nov 22 '19 at 06:35
in my case ,How do I adapt your code, I am saving json file to`d` – Rahul rajan Nov 22 '19 at 06:36
just put p.produce('topic', jd) as you already have json dump in 'jd' variable. – sky Nov 22 '19 at 06:39

score 1 · Answer 3 · edited Nov 27 '21 at 17:41

1

You can send a dict by simply doing like that:

producer = KafkaProducer(
    acks='all',
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: bytes(json.dumps(v, default=str).encode('utf-8'))
)

I call bytes() method before returning the json in the lambda

edited Nov 27 '21 at 17:41

Yaakov Bressler

9,056
2
45
69

answered Nov 27 '21 at 16:54

Hamza ABIDI

11
2

how to send Json file to kafka from Python

3 Answers3