0

I made the direct stream with pyspark kafka. But I couldn't set the value of fromOffset in it. It gives me an exception ClassCastException Long cannot be assigned to int But the problem is we dont have long datatype in python version 3. So how to resolve this issue ? How can I set the fromOffset value without casting it to long in python 3? I am doing this to consume those messages which remained unread when the consumer was dead... I used approach mentioned in this link pySpark Kafka Direct Streaming update Zookeeper / Kafka Offset

Thanks

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Hamza Sheikh
  • 117
  • 3
  • 13

1 Answers1

-1

Please check your spark submit code in case if you are using 0_10 version streaming jar file You will get above error. 0_10 version will not support python. Instead of 0_10 you can use 0_8 Kafka version.

Kafka 0_8 Version will accept the integer offset value.

Could you please refer below document.

https://spark.apache.org/docs/2.2.0/streaming-kafka-integration.html

Muthu M
  • 26
  • 2