Apache spark streaming kafka API vs kinesis API

Asked Oct 18 '18 at 17:23

Active Dec 01 '18 at 08:47

Viewed 221 times

I am having an scala spark application in which I need to switch between streaming from kafka vs kinesis based on the application configuration.

Both the spark API's for kafka streaming (spark-streaming-kafka-0-10_2.11) and kinesis streaming (spark-streaming-kinesis-asl_2.11) returns an InputDStream on creating a stream, but the value types are different.

Kafka stream creating returns InputDStream[ConsumerRecord[String, String]], whereas, Kinesis stream creating returns InputDStream[Array[Byte]]

Is there any API that returns a generic InputDStream irrespective of kafka or kinesis, so that my stream processing can have a generic implementation, instead of having to write separate code for kafka and kinesis.

I tried assigning both the stream to a InputDStream[Any], but that did not work.

Appreciate any idea on how this can be done.

edited Dec 01 '18 at 08:47

Ruslan Ostafiichuk

4,422
6
30
35

asked Oct 18 '18 at 17:23

Biju Gopinathan

You could have Kafka return a `ConsumerRecord[Array[Byte], Array[Byte]]`, but the fact you get one binary blob from Kinesis and a Record object from Kafka can't really be applied in generics – OneCricketeer Oct 18 '18 at 20:07
That makes sense, thanks for the comment. – Biju Gopinathan Oct 24 '18 at 23:32

Apache spark streaming kafka API vs kinesis API

0 Answers0