0

I develop Spark code using Scala APIs on IntelliJ and when I run this I get below error but runs well on the Databricks notebook though.

I am using Databricks Connect to connect from local installation of IntelliJ to the Databricks Spark Cluster. I am connected to the cluster and was able to submit a job from IntelliJ to the Cluster too. AMOF, everything else works except the below piece.

DBConnect is 6.1 , Databricks Runtime is 6.2 Imported the jar file from the cluster (using Databricks-connect get-jar-dir) and set up the SBT project with the jar in the project library

source code:

val sparkSession = SparkSession.builder.getOrCreate()
val sparkContext = sparkSession.sparkContext

import sparkSession.implicits._

val v_textFile_read = sparkContext.textFile(v_filename_path)
v_textFile_read.take(2).foreach(println)

Error:

cannot assign instance of scala.Some to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of
type scala.collection.Seq in instance of org.apache.spark.rdd.HadoopRDD

The reason I use a RDD reader for text is so I can pass this output to a createDataFrame API. As you know, the createdataframe API takes in an RDD and schema as input parameters.

step-1: val v_RDD_textFile_read = sparkContext.textFile(v_filename_path).map(x => MMRSplitRowIntoStrings(x))

step-2: val v_DF_textFile_read = sparkSession.sqlContext.createDataFrame(v_RDD_textFile_read, v_schema) (edited

sarveshseri
  • 13,738
  • 28
  • 47
Aady
  • 54
  • 7
  • `import spark.implicits._` ? – mazaneicha Dec 11 '19 at 22:08
  • I tried importing the implicits library too but that did not work. – Aady Dec 11 '19 at 22:19
  • When you say it does not work... what does actually happen... does it not compile or does it not run ? And what is `v_filename_path` ? Also what is the line causing this error ? – sarveshseri Dec 12 '19 at 09:16
  • you can create spark session in IntelliJ using below code ```implicit val ss: SparkSession = spark implicit val ut: DBUtilsV1 = dbutils``` – Praveen Dec 12 '19 at 09:39

0 Answers0