7

I get below exception when I try to create a dStream within a Function call of Spark.

My call method :

@Override
public JavaRDD<Object> call(JavaRDD<Object> v1) throws Exception {
    Queue<JavaRDD<Object>> queue = new LinkedList<>();
    queue.add(v1);
    JavaDStream<Object> dStream = context.queueStream(queue);
    JavaDStream<Object> newDStream = dStream.map(AbstractProcessor.this);
    final JavaRDD<Object> rdd = context.sparkContext().emptyRDD();
    newDStream.foreachRDD(new SaxFunction<JavaRDD<Object>, Void>() {
        private static final long serialVersionUID = 672054140484217234L;

        @Override
        public Void execute(JavaRDD<Object> object) throws Exception {
            rdd.union(object);
            return null;
        }
    });
    return rdd;
}

Exception :

Caused by: java.lang.IllegalStateException: Adding new inputs, transformations, and output operations after starting a context is not supported
    at org.apache.spark.streaming.dstream.DStream.validateAtInit(DStream.scala:220)
    at org.apache.spark.streaming.dstream.DStream.<init>(DStream.scala:64)
    at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:42)
    at org.apache.spark.streaming.dstream.QueueInputDStream.<init>(QueueInputDStream.scala:29)
    at org.apache.spark.streaming.StreamingContext.queueStream(StreamingContext.scala:513)
    at org.apache.spark.streaming.StreamingContext.queueStream(StreamingContext.scala:492)
    at org.apache.spark.streaming.api.java.JavaStreamingContext.queueStream(JavaStreamingContext.scala:436)

Is there any way I can create a dStream and do operations on it in runtime or I can update DAG after context is started? Thanks in advance.

Amit Kumar
  • 2,685
  • 2
  • 37
  • 72
  • Hi @amit_kumar, do you have any progress with this issue? i have the same... – aironman May 25 '16 at 14:20
  • The answer to my question "Is there any way I can create a dStream and do operations on it in runtime or I can update DAG after context is started?" is no as per https://issues.apache.org/jira/browse/SPARK-2243. – Amit Kumar May 25 '16 at 18:37
  • I'm not sure how that Jira _"Support multiple SparkContexts in the same JVM"_ relates to your question above – Brad Mar 03 '17 at 23:03
  • @Brad I wanted to update DAG after spark context is started or I can create another DAG in same JVM, this is possible if multiple spark context are supported but as per JIRA link SPARK-2243 this will not be done. – Amit Kumar Mar 04 '17 at 08:41
  • @AmitKumar is there any update? – Akbar Noto May 23 '19 at 05:42

0 Answers0