How to convert Array[String] to Array[Any] in Spark/Scala

Question

I am trying to generate sourceIds for the parallelPersonalizedPageRank algorithm inside Graphframes and call the algoirthm as following:

val PPRIdCS = studentCS.select("id").collect.map(row => row.getString(0))
val ranksCS = studentGraph
  .parallelPersonalizedPageRank
  .resetProbability(0.15)
  .maxIter(10)
  .sourceIds(PPRIdCS)
  .run()

The error information I got is as following:

Message: <console>:46: error: type mismatch;
found   : Array[String]
required: Array[Any]
Note: String <: Any, but class Array is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 
3.2.10).sourceIds(PPRIdCS)

I could not figure out what is the way to cast a String type to Any type, or a way to map String to Any while generating PPRIdCS. Thanks!

I tried this val PPRIdCS = studentCS.select("id").collect.map(row => row.getString(0)).map(_.asInstanceOf(Any)). But get the error saying: Name: Unknown Error Message: :47: error: T0 does not take parameters val PPRIdCS = studentCS.select("id").collect.map(row => row.getString(0)).map(\_.asInstanceOf(Any)) — Guanghua Shu, Apr 03 '18 at 22:49
`val arr = Array("a", "b") val asAny: Array[Any] = arr.map(_.asInstanceOf[Any])` — Jordan Cutler, Apr 03 '18 at 22:50
Thank you Jordan. I made an error of using () for Any type. What you suggested works. The one in the answer seem to be a more direct way to define the type as well. — Guanghua Shu, Apr 03 '18 at 23:24
@GuanghuaShu If one of the proposed solutions worked, then please take a few seconds to [accept an answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work). — Andrey Tyukin, Apr 04 '18 at 11:49

Jordan Cutler · Accepted Answer · 2018-04-04T00:05:26.260

5

As noted from my comment, you can map to .asInstanceOf[Any].

Example:

val arr = Array("a", "b") 
val asAny = arr.map(_.asInstanceOf[Any])

This also appears to work...

val asAny = arr.asInstanceOf[Array[Any]]

Now doing this is assuming you for some reason do not want to specify the type explicitly as noted by the other answer.

edited Apr 04 '18 at 00:05

answered Apr 03 '18 at 23:26

Jordan Cutler

562
4
14

score 2 · Answer 2 · answered Apr 03 '18 at 22:54

2

Just be explicit about the types:

val PPRIdCS: Array[Any] = studentCS.select("id").collect.map(row => row.getString(0))

or better

val PPRIdCS: Array[Any] = studentCS.select("id").collect.map(row => row(0))

answered Apr 03 '18 at 22:54

user9593763

21
1

2

Wouldn't it be faster if you swapped `map` and `collect`? The `map` looks as if it could throw away quite a few unnecessary row entries, then there would be no need to send them to master node. – Andrey Tyukin Apr 03 '18 at 23:22
1

@AndreyTyukin I think swap map and collect is a good idea. In this particular case it may not matter since "id" has only one element. – Guanghua Shu Apr 05 '18 at 02:03

How to convert Array[String] to Array[Any] in Spark/Scala

2 Answers2