1

I am using spark version 2.2. I am trying to create a dataframe with 1 column as MapType.

I have tried following things for that:

scala> val mapSeq = Seq((Map(1-> 2, 11-> 22))).toDF("num")
<console>:23: error: value toDF is not a member of Seq[scala.collection.immutable.Map[Int,Int]]
       val mapSeq = Seq((Map(1-> 2, 11-> 22))).toDF("num")

and

scala> spark.createDataFrame(List(Map("a" -> "b")), StructType(MapType(StringType, StringType)))

error: overloaded method value apply with alternatives:
  (fields: Array[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType <and>
  (fields: java.util.List[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType <and>
  (fields: Seq[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType
 cannot be applied to (org.apache.spark.sql.types.MapType)
       spark.createDataFrame(List(Map("a" -> "b")), StructType(MapType(StringType, StringType)))

Both the methods throws error.

I tried searching and found this medium blog - https://medium.com/@mrpowers/working-with-spark-arraytype-and-maptype-columns-4d85f3c8b2b3 But it blog there is some older version of spark used.

Any Ideas on how can I create this?

dijksterhuis
  • 1,225
  • 11
  • 25
RIYA SINGHAL
  • 51
  • 6
  • [This](https://stackoverflow.com/a/46779193/2639647) answer seems to work. – Travis Hegner Mar 13 '19 at 15:06
  • Possible duplicate of [How to create a Dataset of Maps?](https://stackoverflow.com/questions/46778659/how-to-create-a-dataset-of-maps) – koiralo Mar 13 '19 at 15:42
  • It doesn't look like a duplicate. The answer tells about datasets and not dataframes. Is there any way to create dataframes of Maps? – RIYA SINGHAL Mar 14 '19 at 06:36

0 Answers0