Putting two placeholders inside flatMap in Spark Scala to create Array

Question

I am applying flatMap on a scala array and create another array from it:

val x = sc.parallelize(Array(1,2,3,4,5,6,7))
val y = x.flatMap(n => Array(n,n*100,42))
println(y.collect().mkString(","))
1,100,42,2,200,42,3,300,42,4,400,42,5,500,42,6,600,42,7,700,42

But I am trying to use placeholder "_" in the second line of the code where I create y in the following way:

scala> val y = x.flatMap(Array(_,_*100,42))
<console>:26: error: wrong number of parameters; expected = 1
       val y = x.flatMap(Array(_,_*100,42))
                              ^

Which is not working. Could someone explain what to do in such cases if I want to use placeholder?

score 2 · Accepted Answer · answered Aug 17 '15 at 17:59

In scala, the number of placeholders in a lambda indicates the cardinality of the lambda parameters.

So the last line is expanded as

val y = x.flatMap((x1, x2) => Array(x1, x2*100, 42))

Long story short, you can't use a placeholder to refer twice to the same element.

You have to use named parameters in this case.

val y = x.flatMap(x => Array(x, x*100, 42))

score 0 · Answer 2 · answered Aug 17 '15 at 18:01

You can only use _ placeholder once per parameter. (In your case, flatMap method takes single argument, but you are saying -- hey compiler, expect two arguments which is not going to work)

val y = x.flatMap(i => Array(i._1, i._2*100,42))

should do the trick.

val y = x.flatMap { case (i1, i2) => Array(i1, i2*100,42) }

should also work (and probably more readable)

Putting two placeholders inside flatMap in Spark Scala to create Array

2 Answers2