Scala access Map[String, Any] with nested data structures

Question

I have s simple map like

  val parameters: Map[String, Any] = Map("digits" -> Seq(1, 2, 3, 4, 5, 6, 7, 8,

an want to multiply each number by 3 like shown below

class PrintMap extends App {
  val conf: SparkConf = new SparkConf()
    .setAppName("sparkApiSample")
    .setMaster("local[*]")

  val session: SparkSession = SparkSession
    .builder()
    .config(conf)
    .getOrCreate()

  val parameters: Map[String, Any] = Map("digits" -> Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 0))

  val numbers: Seq[Int] = parameters("digits").asInstanceOf[Seq[Int]]
  val rdd = session.sparkContext.parallelize(numbers)

  val result = Map("result" -> rdd.map(x => x * 3).collect())

  // want to "access / print the contents of the Array at result
  result.get("result") match {
    case Some(x) => x.asInstanceOf[Seq[Any]].foreach(println)
    case None => println("error occurred")
  }

Why does it result in the following exception and how could I actually access the map? java.lang.ClassCastException: [I cannot be cast to scala.collection.Seq

You are using unsafe code -> asInstanceOf. Try pattern matching with types: http://stackoverflow.com/questions/15655817/scala-multiple-type-pattern-matching — Pavel, Dec 01 '16 at 10:14
What do you mean?. You are using unsafe code. Add method/code to do it in a safe way. — Pavel, Dec 01 '16 at 10:17
Sure. I am aware that x.asInstanceOf[Seq[Any]] is unsafe. But I an not sure what `case ClassName` to choose for this `I` class. — Georg Heiler, Dec 01 '16 at 10:19
https://spark.apache.org/docs/0.8.1/api/core/org/apache/spark/rdd/RDD.html hope this helps. — Pavel, Dec 01 '16 at 10:23

A Spoty Spot · Accepted Answer · 2016-12-01T10:36:16.443

3

Collect on an RDD returns an Array. Array does not extend Seq. So your x cannot be cast to a Seq.

eg.)

Array(2).asInstanceOf[Seq[Int]]

Throws the same exception.

Instead your result should be of type: Map[String, Array[Int]]

So instead just use x.toSeq isntead of x.asInstanceOf[Seq[Int]]

EDIT: "[I" in your stack means Array of Int.

It occurred to me as I was writing this, I'm guessing the reason you're using any is because each your have a bunch of different parameters and return types in your array. If this is the case it'd be handy see a slightly more complete example.

edited Dec 01 '16 at 10:36

answered Dec 01 '16 at 10:22

A Spoty Spot

746
6
19

Thanks `result.get("result").toSeq.asInstanceOf[List[Array[Int]]].flatten.foreach(println)` works fine. – Georg Heiler Dec 01 '16 at 10:32
2

Glad to see you got it working. But honestly if your casting this much you're probably doing something'wrong' – A Spoty Spot Dec 01 '16 at 10:38

Scala access Map[String, Any] with nested data structures

1 Answers1