Kafka reading txt file as single letters rather than words

Question

I'm using the KafkaClient in python's pykafka. I'm trying to read a text file and produce its lines to a topic then read it by a consumer. However on running it only reads individual letters in the message not the words or the lines of the text file. What am I doing wrong?

my producer is

from pykafka import KafkaClient

text = open('filename.txt','r').read()
text = text.split()
client = KafkaClient(hosts='localhost:9099')
topic = client.topics['topic']
producer = topic.get_sync_producer()
for i in text:
     producer.produce(i.encode('ascii'))

my consumer is

from pykafka import KafkaClient
client = KafkaClient(hosts='localhost:9099')
topic = client.topics['topic']
consumer = topic.get_simple_consumer()
for message in consumer:
    if message is not None:
        print(message.offset, message.value.decode())

Would appreciate some pointers. I'm wondering if this is the best way to read a text file and run it through kafka.

Your problem seems to be "how to read file line by line", unrelated to kafka. Did you try pykafka libraries with the solution your wrote in the answer? — Rishabh Sharma, Aug 28 '20 at 02:25

score 0 · Answer 1 · answered Aug 27 '20 at 13:10

I got the desired output by switching libraries to the kafka libraries.

Producer

from kafka import KafkaProducer

producer = 
KafkaProducer(bootstrap_servers='localhost:9099')

with open('filename.txt') 
as f:
    for line in f:
      
     producer.send('topic',line.encode('ascii'))

 producer.flush()
 producer.close()

Consumer

from kafka import KafkaConsumer

consumer =

KafkaConsumer('topic',bootstrap_servers='localhost:9099')

for msg in consumer:
    print(msg.value.decode())

Kafka reading txt file as single letters rather than words

1 Answers1