0

Am trying to send messages from python to kafka consumer. But am getting error as ERROR Unknown error when running consumer: org.apache.kafka.common.errors.SerializationException: Unknown magic byte! Python is properly getting the data from twitter api, but unable to send the messages to the consumer. Any suggestion would be helpful.

Code:

from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
from kafka import KafkaProducer

access_token = "xxxx"
access_token_secret = "xxxx"
consumer_key = "xxxx"
consumer_secret = "xxxx"

class StdOutListener(StreamListener):
    def on_data(self, data):
        producer.send("twitter", data.encode('utf-8'))
        print (data)
        return True
    def on_error(self, status):
        print (status)
        
producer = KafkaProducer(bootstrap_servers='127.0.0.1:9092')
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
stream.filter(track="Bitcoin")

Consumer:

kafka-avro-console-consumer --bootstrap-server 127.0.0.1:9092 --topic twitter

I have tried providing the properties as well,

afka-avro-console-consumer --bootstrap-server 127.0.0.1:9092 --topic twitter --property print.key=true --property print.value=true --value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer --key-deserializer org.apache.kafka.common.serialization.StringDeserializer

Error:

[2020-08-14 07:23:52,305] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.SerializationException: Error deserializing Protobuf message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
Jim Macaulay
  • 4,709
  • 4
  • 28
  • 53

1 Answers1

3

Error seems to think it's getting Proto messages

Error deserializing Protobuf message

But you are using kafka-avro-console-consumer

Meanwhile, your code is sending utf-8 encoded strings, so you don't need any Confluent tools, just kafka-console-consumer

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245