addition of two dataframe integer values in Scala/Spark

Question

So I'm new to both Scala and Spark so it may be kind of a dumb question... I have the following code :

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

val df = sc.parallelize(List(1,2,3)).toDF();
df.foreach( value => println( value(0) + value(0) ) );

error: type mismatch;
 found   : Any
 required: String

What is wrong with it? How do I tell "this is an integer not an any"? I tried value(0).toInt but "value toInt is not a member of Any". I tried List(1:Integer, 2:Integer, 3:Integer) but I can not convert into a dataframe afterward...

score 0 · Accepted Answer · edited May 23 '17 at 12:25

Spark Row is an untyped container. If you want to extract anything else than Any you have to use typed extractor method or pattern matching over the Row (see Spark extracting values from a Row):

df.rdd.map(value => value.getInt(0) + value.getInt(0)).collect.foreach(println)

In practice there should be reason to extract these values at all. Instead you can operate directly on the DataFrame:

df.select($"_1" + $"_1")

addition of two dataframe integer values in Scala/Spark

1 Answers1