how to get the prediction of a model in pyspark

Question

i have developed a clustering model using pyspark and i want to just predict the class of one vector and here is the code

spark = SparkSession.builder.config("spark.sql.warehouse.dir", 
            "file:///C:/temp").appName("Kmeans").getOrCreate()
 vecAssembler = VectorAssembler(inputCols=FEATURES_COL, outputCol="features")
 df_kmeans = vecAssembler.transform(df).select('LCLid', 'features')
 k = 6
 kmeans = KMeans().setK(k).setSeed(1).setFeaturesCol("features")
 model = kmeans.fit(df_kmeans)
 centers = model.clusterCenters()
 predictions = model.transform(df_kmeans)
 transformed = model.transform(df_kmeans).select('LCLid', 'prediction')
 rows = transformed.collect()

say that i have a vector of features V and i want to predict in which class it belongs i tried a method that i found in this link http://web.cs.ucla.edu/~zhoudiyu/tutorial/ but it doesn't work since i'm working with SparkSession not in sparkContext

Answer 1

I see that you dealt with the most basic steps in your model creation, what you still need is to apply your k-means model on the vector that you want to make the clustering on (like what you did in line 10) then get your prediction, I mean what you have to do is to reDo the same work done in line 10 but on the new vector of features V. To understand this more I invite you to read this posted answer in StackOveflow: KMeans clustering in PySpark. I want to add also that the problem in the example that you are following is not due to the use of SparkSession or SparkContext as those are just an entry point to the Spark APIs, you can also get access to a sparContext through a sparkSession since it is unified by Databricks since Spark 2.0. The pyspark k-means is like the Scikit learn the only difference is the predefined functions in spark python API (PySpark).

Answer 2

You can call the predict method of the kmeans model using a Spark ML Vector:

from pyspark.ml.linalg import Vectors

model.predict(Vectors.dense([1,0]))

Here [1,0] is just an example. It should have the same length as your feature vector.