Error with RDD[Vector] in function parameter

Question

I am trying to define a function in scala to iterate on it with Spark. Here is my code :

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext
import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.ml.clustering.KMeans
import org.apache.spark.mllib.linalg.Vectors

import org.apache.spark.ml.feature.VectorIndexer
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.rdd._

    val assembler = new VectorAssembler()
          .setInputCols(Array("feature1", "feature2", "feature3"))
          .setOutputCol("features")
val assembled = assembler.transform(df)

// measures the average distance to centroid, for a model built with a given k.

def clusteringScore(data: RDD[Vector],k:Int) = {

val kmeans = new KMeans()
    .setK(k)
    .setFeaturesCol("features")
    .setPredictionCol("prediction")
    val model = kmeans.fit(data)

  val WSSSE = model.computeCost(data)   println(s"Within Set Sum of Squared Errors = $WSSSE")

}

(5 to 40 by 5).map(k => (k, clusteringScore(assembled, k))).
      foreach(println)

With this code I get this error :

type Vector takes type parameters

I don't know what means this error...

score 9 · Accepted Answer · answered May 13 '16 at 15:26

You are not showing your imports, but you are probably importing Scala standard collections' Vector(this one takes a type parameter, e.g. Vector[Int]) instead of the SparkML Vector, which is a different type and you should import like this:

import org.apache.spark.mllib.linalg.Vector

Error with RDD[Vector] in function parameter

1 Answers1