Json producer in kafka-python

Question

I am trying a simple kafka producer first time which will take data from json file record by record. But I am getting error.

My Json File(test.json):

{
  "states": 
  [
    {
      "name": "Alabama",
      "abbreviation": "AL"
    },
    {
      "name": "Alaska",
      "abbreviation": "AK"
    }
  ]
}

My producer class:

import json
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
print('Producer created..............')

with open('/home/ravi/test.json') as f:
    data = json.load(f)

    for state in data['states']:
        producer.send('ECJson', json.dump(state))

But I am getting error:

Producer created..............
Traceback (most recent call last):
  File "prodECJson.py", line 10, in <module>
    producer.send('ECJson', json.dump(state))
TypeError: dump() missing 1 required positional argument: 'fp'

Rémi Chauvenne · Answer 1 · 2021-03-09T18:26:00.453

json.dump will write on disk. What you want is json.dumps that will create a string from the given object. The send function of KafkaProducer expects bytes so you have to encode the string too. You can also specify the serializer in the KafkaProducer object directly.

For example:

import json
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092', 
                         value_serializer=lambda m: json.dumps(m).encode("utf-8"))
print('Producer created..............')

with open('/home/ravi/test.json') as f:
    data = json.load(f)

    for state in data['states']:
        producer.send('ECJson', state)

Json producer in kafka-python

1 Answers1