0

I am creating a javaPairRDD and saving it to sequenceFileFormat with apache-spark. Spark version is 2.3. I am running this on normal 4 node cluster and path is also normal hdfs path. I am doing it using spark code (Java):

JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD <Integer> jr = sc.parallelize(Arrays.asList(1,2,3,4,5));
JavaPairRDD<NullWritable, Integer> outputData = jr.mapToPair( p -> { 
            return new Tuple2<>(NullWritable.get(), p);
        });
outputData.saveAsHadoopFile("hdfs://master:54310/user/output12",
                NullWritable.class, IntWritable.class, SequenceFileOutputFormat.class);
sc.close();

But when i try to run the code I get following exception

org.apache.spark.SparkException: Task failed while writing rows
DAVID_ROA
  • 309
  • 1
  • 3
  • 18
  • Many reasons for such exception are possible, more information is required, please check logs. – pasha701 Sep 12 '18 at 11:47
  • when I export program to jar file and run job from cluster this exception does not occur but when I run it from IDE (from remote computer) this exception occur. – DAVID_ROA Sep 18 '18 at 05:06

0 Answers0