Converting datatypes in Spark/Scala

Question

I have a variable in scala called a which is as below

scala> a
res17: Array[org.apache.spark.sql.Row] = Array([0_42], [big], [baller], [bitch], [shoe] ..)

It is an array of lists which contains a single word.

I would like to convert it to a single array consisting of sequence of strings like shown below

Array[Seq[String]] = Array(WrappedArray(0_42,big,baller,shoe,?,since,eluid.........

Well the reason why I am trying to create an array of single wrapped array is I want to run word2vec model in spark using MLLIB.

The fit() function in this only takes iterable string.

scala> val model = word2vec.fit(b)
<console>:41: error: inferred type arguments [String] do not conform to method fit's type parameter bounds [S <: Iterable[String]]

can you provide an example input and an example output how it should be — zenwraight, Dec 29 '17 at 17:16
Example input is a Array([a],[b],[c],[d]) and output should be Array(wrappedarray(a,b,,c,d)) — vish, Dec 29 '17 at 17:21
Possible duplicate of [Spark extracting values from a Row](https://stackoverflow.com/questions/33007840/spark-extracting-values-from-a-row) — philantrovert, Dec 29 '17 at 18:31

Leo C · Answer 1 · 2017-12-29T19:38:18.033

The sample data you're listing is not an array of lists, but an array of Rows. An array of a single WrappedArray you're trying to create also doesn't seem to serve any meaningful purpose.

If you want to create an array of all the word strings in your Array[Row] data structure, you can simply use a map like in the following:

val df = Seq(
  ("0_42"), ("big"), ("baller"), ("bitch"), ("shoe"), ("?"), ("since"), ("eliud"), ("win")
).toDF("word")

val a = df.rdd.collect
// a: Array[org.apache.spark.sql.Row] = Array(
//   [0_42], [big], [baller], [bitch], [shoe], [?], [since], [eliud], [win]
// )

import org.apache.spark.sql.Row

val b = a.map{ case Row(w: String) => w }
// b: Array[String] = Array(0_42, big, baller, bitch, shoe, ?, since, eliud, win)

[UPDATE]

If you do want to create an array of a single WrappedArray, here's one approach:

val b = Array( a.map{ case Row(w: String) => w }.toSeq )
// b: Array[Seq[String]] = Array(WrappedArray(
//   0_42, big, baller, bitch, shoe, ?, since, eliud, win
// ))

Well the reason why I am trying to create an array of single wrapped array is I want to run word2vec model in spark using MLLIB. The fit() function in this only takes iterable string. I could create val b ,but I cannot run the model using it....I get the following error scala> val model = word2vec.fit(b) :41: error: inferred type arguments [String] do not conform to method fit's type parameter bounds [S <: Iterable[String]] val model = word2vec.fit(b) — vish, Dec 29 '17 at 19:24
@vish, `a` as in your original sample data is not a `RDD` but an `Array[Row]`. Using my sample data, you'll need to `collect` (i.e. df.rdd.collect) in order to get to the exact `Array[Row]` type, in which `toSeq` is applicable. — Leo C, Dec 29 '17 at 20:35

vish · Answer 2 · 2018-01-03T22:30:01.423

0

I finally got it working by doing the following

val db=a.map{ case Row(word: String) => word }
val model = word2vec.fit( b.map(l=>Seq(l)))

edited Jan 03 '18 at 22:30

answered Dec 29 '17 at 20:56

vish

67
3
10

Where does `f` come from? – Leo C Dec 29 '17 at 21:23

Converting datatypes in Spark/Scala

2 Answers2