Convert entry hocon to dataframe

Question

I have this entry "channels" in hocon format:

param {
  channels = [
    {
      name = "MainPhone"
      id = 1
    }
    {
      name = "OpPhone"
      id = 2
    }
    {
      name = "Fax"
      id = 3
    }
  ]
}

What I need is to convert the entry channels into a dataframe, is that possible? I was trying with the following code:

val ch = confParam.getConfigList("param.channels")
case class Provider(id: String, name: String)
val l = ch.map(conf => Provider(conf.getString("id"), conf.getString("name"))).toList
val df = l.toDF()

but I get the error:

"Error: (28, 20) value toDF is not a member of List [Provider] val df = list.toDF ()"

see my answer to https://stackoverflow.com/questions/51042048/parsing-scala-json-into-dataframe/51046795#51046795 — Arnon Rotem-Gal-Oz, Jun 27 '18 at 18:59

score 0 · Answer 1 · answered Jun 28 '18 at 03:11

To use toDF() in Scala you need to import spark.implicits in your code. spark here refers to to a SparkSession object, for example in Spark 2.0+:

val spark = SparkSession.builder().getOrCreate()

Then do the import:

import spark.implicits._

This will allow you to use functions such as toDF, toDS, etc.

Convert entry hocon to dataframe

1 Answers1