Linear regression with SGD using pyspark.ml.linearegression

Question

I'm using the LinearRegression model in the Spark ML for prediction.

import pyspark.ml.regression.LinearRegression

featureassembler = VectorAssembler(inputCols=[‘Year’, ‘Present_Price’, 
                                              ‘Kms_Driven’, ‘Owner’], 
                                   outputCol=’features’)

output = featureassembler.transform(df)
data = output.select('features', 'Selling_Price')

# Initializing a Linear Regression model
ss = LinearRegression(featuresCol='features', labelCol='Selling_Price')

I want to test the linear regression with SGD(Stochastic Gradient Descent.) but pyspark.ml does not propose any linearregressionwithSGD like mllib. Also, when accessing the mllib linear regressionwithSGD i found that it Deprecated since version 2.0.0.

How can i use ml for linear regression with SGD. Is there any parameter that i can use for that?

Answer 1

Instead of ml you can use mllib:

from pyspark.mllib.regression import LabeledPoint, LinearRegressionWithSGD, LinearRegressionModel

Here is the documentation: https://spark.apache.org/docs/1.6.1/mllib-linear-methods.html