how to send JSON object to kafka from python client

Question

I have a simple JSON object like the following

d = { 'tag ': 'blah',
  'name' : 'sam',
  'score': 
    {'row1': 100,
      'row2': 200
     }
}

The following is my python code which is sending messages to Kafka

from kafka import SimpleProducer, KafkaClient
import json 

# To send messages synchronously
kafka = KafkaClient('10.20.30.12:9092')
producer = SimpleProducer(kafka)
jd = json.dumps(d)
producer.send_messages(b'message1',jd)

I see in the storm logs that the message is being received but its throwing Transformation null for tuple { json structure in here } not sure what needs to be done in order to fix this ?..

that doesn't work. d will be a dictionary. I need to convert that to string or bytes — Rahul, Aug 05 '15 at 18:35
What do you mean by "throwing Transformation null for tuple"? — BAE, Aug 06 '15 at 20:11
@Rahul Can you please Post the solution.... I need to send JSON object to kafka. — yoku2010, Sep 30 '15 at 07:01

vinit kantrod · Answer 1 · 2020-03-23T17:17:31.570

7

Kafka expects the values in bytes

b`some json message`

Here is my simple Kafka producer which sends the message to Kafka server.

import json
from bson import json_util

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')

for i in range(10):
    data = { 'tag ': 'blah',
        'name' : 'sam',
        'index' : i,
        'score': 
            {'row1': 100,
             'row2': 200
        }
    }   
    producer.send('orders', json.dumps(data, default=json_util.default).encode('utf-8'))

Here json.dumps() convert json into string and encode('utf-8') converts string into byte array.

edited Mar 23 '20 at 17:17

answered Feb 24 '18 at 18:56

vinit kantrod

1,306
15
16

Should it not be `producer.send('orders', json.dumps(data, default=json_util.default).encode('utf-8'))` ("d" should be "data") – Alfa Bravo Jan 03 '20 at 10:27
1

In my case it worked when I added `producer.flush()` in the end of the script – EuberDeveloper Jul 02 '22 at 21:48

score 2 · Accepted Answer · answered Sep 30 '15 at 07:25

2

The below is my code for producer to kafka. The only thing i did differently was to use yaml.safe_load to load the json content. It returns the contents as strings instead of unicode. The following is the snippet

with open('smaller_test_prod.txt') as f:
    for line in f:
        d = yaml.safe_load(line)
        jd = json.dumps(d)
        producer.send_messages(b'zeus_metrics',jd)

In here every line is a json data stored in a file.

answered Sep 30 '15 at 07:25

Rahul

11,129
17
63
76

Why would you use YAML to parse JSON? – OneCricketeer Nov 22 '19 at 06:53
Wish I can downvote this. You can convert the string to utf-8 instead – Daniel Kurniadi Feb 11 '20 at 09:00

how to send JSON object to kafka from python client

2 Answers2

Linked