Spark Streaming integration for Kafka. Direct Stream approach provides simple parallelism, 1:1 correspondence between Kafka partitions and Spark partitions, and access to offsets and metadata.
Questions tagged [spark-streaming-kafka]
250 questions
0
votes
1 answer
Scala Spark streaming kafka
I have created a sample topic in kafka and i am trying to consume the content in spark using below script:
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka._
import…
Aish Mahesh
- 171
- 1
- 2
- 10
0
votes
1 answer
Spark Streaming + Kafka integration
I try to integrate spark and kafka in Jupyter notebook by using pyspark. Here is my work environment.
Spark version: Spark 2.2.1
Kafka version: Kafka_2.11-0.8.2.2
Spark streaming kafka jar: spark-streaming-kafka-0-8-assembly_2.11-2.2.1.jar
I added a…
Daniel Lee
- 45
- 1
- 10
0
votes
1 answer
Kafka Spark Streaming LocationStrategies java class def not found exception
I am trying to integrate the kafka message broker and spark and facing an issue saying
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka010/LocationStrategies
Below is the java spark code
package…
Gowtham Kesa
- 99
- 2
- 18
0
votes
0 answers
Spark streaming (Spark 1.6) with CommitAsync API of Kafka 0.10
We need to use the commitAsync API of Kafka consumer. It's available with Kafka 0.9 and 0.10 version.
Can use Spark streaming 1.6 version with Kafka 0.10 ?
As per the blogs and sites I have read, Kafka 0.10 is only used/supported by spark streaming…
0
votes
1 answer
kafka streaming or spark streaming
Am using now kafka in Python.
Was wondering if Spark Kafka is needed or can we use just use kafka
through pyKafka.
My concern was Spark creates overhead (pyspark) in the process,
and if we don't use any spark functions, just Kafka streaming is…
tensor
- 3,088
- 8
- 37
- 71
0
votes
1 answer
sbt unresolved dependency for spark streaming Kafka integration
I want to work with the Kafka integration for Spark streaming. I use Spark version 2.0.0.
But I get a unresolved dependency error ("unresolved dependency: org.apache.spark#spark-sql-kafka-0-10_2.11;2.0.0: not found").
How can I accesss this…
ngi
- 146
- 1
- 8
0
votes
1 answer
java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils
I am using following pom.xml to build my project. When i run the resultant jar with spark submit it gives the following error.
java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils
However if I include --packages…
Swapnil
- 159
- 1
- 11
0
votes
2 answers
does pyspark support spark-streaming-kafka-0-10 lib?
my kafka cluster version is 0.10.0.0, and i want to use pyspark stream to read kafka data. but in Spark Streaming + Kafka Integration Guide, http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
there is no python code…
kula
- 71
- 1
- 4
0
votes
0 answers
Integrating Kafka version 2.11-0.10.0.1 with spark streaming ver 2.1.1
I'm trying to run KafkaWordCount example in spark streaming using Spark version 2.1.1 in standalone cluster mode. As the kafka version on the server that I'm trying to integrate with is 2.11-0.10.0.1 . According to…
Mahendra Singh Meena
- 578
- 2
- 8
0
votes
1 answer
Spark streaming and kafka integration
I'm using kafka and spark streaming for a project programmed in python. I want to send data from kafka producer to my streaming program. It's working smoothly when i execute the following command with the dependencies specified:
./spark-submit…
Akhilesh
- 83
- 10
0
votes
2 answers
Spark streaming with Kafka connector stopping
I am beginning using Spark streaming. I want to get a stream from Kafka with a sample code I found on the Spark documentation : https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html
Here is my code :
object SparkStreaming {
…
ImbaBalboa
- 851
- 9
- 23
0
votes
1 answer
Spark streaming and Kafka intergration
I'm new to Apache Spark and I've been doing a project related to sentiment analysis on twitter data which involves spark streaming and kafka integration. I have been following the github code (link provided below)…
Amal Sailendran
- 341
- 1
- 2
- 16
0
votes
2 answers
Getting java.lang.NoClassDefFoundError: kafka/serializer/StringDecoder Exception while streaming kafka from Spark streaming
I am trying to read the kafka streaming data from spark streaming application; while in the process of reading data I am getting following exception:
16/12/24 11:09:05 INFO storage.BlockManagerMaster: Registered BlockManager
Exception in thread…
kumar
- 207
- 1
- 3
- 9
0
votes
1 answer
Spark Streaming Kafka consumer doesn't like DStream
I am using Spark Shell (Scala 2.10 and Spark Streaming org.apache.spark:spark-streaming-kafka-0-10_2.10:2.0.1) to test a Spark/Kafka consumer:
import org.apache.kafka.clients.consumer.ConsumerRecord
import…
smeeb
- 27,777
- 57
- 250
- 447
0
votes
1 answer
spark kafka streaming Error - " java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils
I am writing a simple kafka - spark streaming code in eclipse to consume the messages from kafka broker using spark streaming. Below is the code, i receive the error when i try to run the code from eclipse.
I also made sure the dependency jars are…
keerthi
- 23
- 1
- 8