Implement case class inside a class

Question

I am using the below code to run in Qubole Notebook and the code is running successfully.

case class cls_Sch(Id:String, Name:String)
class myClass { 
    implicit val sparkSession = org.apache.spark.sql.SparkSession.builder().enableHiveSupport().getOrCreate()
    sparkSession.sql("set spark.sql.crossJoin.enabled = true")
    sparkSession.sql("set spark.sql.caseSensitive=false")   
    import sparkSession.sqlContext.implicits._
    import org.apache.hadoop.fs.{FileSystem, Path, LocatedFileStatus, RemoteIterator, FileUtil}
    import org.apache.hadoop.conf.Configuration 
    import org.apache.spark.sql.DataFrame

    def my_Methd() {                

        var my_df = Seq(("1","Sarath"),("2","Amal")).toDF("Id","Name")      

        my_df.as[cls_Sch].take(my_df.count.toInt).foreach(t => {            

            println(s"${t.Name}")

        })              
    }
}
val obj_myClass = new myClass()
obj_myClass.my_Methd()

However when I run in the same code in Qubole's Analyze, I am getting the below error.

When I take out the below code, its running fine in Qubole's Anlayze.

my_df.as[cls_Sch].take(my_df.count.toInt).foreach(t => {            

            println(s"${t.Name}")

        })

I believe somewhere I have to change the usage of case class.

I am using Spark 2.3.

Can someone please let me know how to solve this issue.

Please let me know if you need any other details.

You mean to say "sparkSession.implicits._" instead of "sparkSession.sqlContext.implicits._"? @EmiCareOfCell44 — Sarath Subramanian, Jul 10 '19 at 14:51
Yes, the error says that the product encoders(case classes) are available in that package — Emiliano Martinez, Jul 10 '19 at 14:54
I tried that way also. Its throwing the same error @EmiCareOfCell44 — Sarath Subramanian, Jul 10 '19 at 14:55

Emiliano Martinez · Answer 1 · 2022-08-30T08:39:46.703

1

For any reason the kernel finds problems when working with dataset. I made two tests that worked with Apache Toree:

edited Aug 30 '22 at 08:39

answered Jul 10 '19 at 15:46

Emiliano Martinez

4,073
2
9
19

Thanks Emi. I will check this and let you know tomorrow – Sarath Subramanian Jul 10 '19 at 16:38

Prasad Sogalad · Answer 2 · 2019-11-21T04:58:32.813

1

All you have to do is have the import spark.implicits._ inside the my_Methd() function.

def my_Methd() {   

    import spark.implicits._     

    var my_df = Seq(("1","Sarath"),("2","Amal")).toDF("Id","Name")      

    my_df.as[cls_Sch].take(my_df.count.toInt).foreach(t => {            

        println(s"${t.Name}")

    })              
}

edited Nov 21 '19 at 04:58

answered Nov 21 '19 at 00:07

Prasad Sogalad

131
1
5

Implement case class inside a class

2 Answers2