How to gracefully stop a spark dStream process

Question

I'm trying to read data from a kafka stream, process it and save it to a report. I'd like to run this job once a day. I'm using dStreams. Is there an equivalent of trigger(Trigger.Once) in dStreams I could use for this scenario. appreciate suggestions and help.

def main(args: Array[String]) {
val spark = ...
val ssc =  new StreamingContext(sc, Seconds(jobconfig.getLong("batchInterval")))
       val kafkaStream =
      KafkaUtils.createDirectStream[String, String](ssc, LocationStrategies.PreferConsistent, ConsumerStrategies.Subscribe[String, String](Array(jobconfig.getString("topic")), kafkaParams))
      kafkaStream.foreachRDD(rdd => {
.
.
.
.
}
    sqlContext.clearCache()
    ssc.start()
    ssc.awaitTermination()
}

score 0 · Answer 1 · answered Dec 13 '19 at 20:16

0

Anil,

According to the documentation you should use the spark.streaming.stopGracefullyOnShutdown parameter : https://spark.apache.org/docs/latest/configuration.html

It says :

If true, Spark shuts down the StreamingContext gracefully on JVM shutdown rather than immediately.

answered Dec 13 '19 at 20:16

baitmbarek

2,440
4
18
26

1

Thank you @baitmbarek - Unfortunately, this is not what I was looking for. I figured out a way to do this by stopping the spark context when the rdd is empty. – Anil Dec 16 '19 at 16:22
@Anil could you please provide more info on what worked for you (Code would be ideal) or documentation. I am facing the same issue. – deltascience Sep 15 '21 at 20:38

How to gracefully stop a spark dStream process

1 Answers1