Turning a continuous variable into categorical in Spark

Question

I am trying to turn some continuous variables into categorical variables in order to apply some ML algorithms to them and I want to make categories like from 6:00 to 12:00 --> "Morning" or dates in a format like ddMM to "Summer" or whatever. These variables are already casted to integers. Like the recode function in R, I think.

+----------+
|CRSDepTime|
+----------+
|       745|
|      1053|
|      1915|
|      1755|
|       832|
|       630|
|       820|
|       945|
|      1245|
|      1645|
|       620|
|      1125|
|      2045|
|      1340|
|      1540|
|       730|
|      1145|
|       525|
|       630|
|      1520|
+----------+

I solved this issue with this sentence!!

df = df.withColumn("Season", when(df("Month") >= 12 and df("Month") <=3, "Fall")
  .when(df("Month") >= 4 and df("Month") <= 6, "Spring")
  .when(df("Month") >= 7 and df("Month") <= 9, "Summer").otherwise("Autumm"))

score 10 · Accepted Answer · answered Dec 04 '17 at 20:05

There are two Transformers which can be used to convert continuous variables to categorical ones:

Bucketizer
QuantileDiscretizer

Bucketizer takes splits so it can be used here:

import org.apache.spark.ml.feature._

val df = Seq(
  745, 1053, 1915, 1755, 832, 630, 820, 945,
  1245, 1645, 620, 1125, 2045, 1340, 1540, 730,
  1145, 525, 630, 1520
).toDF("CRSDepTime")

val bucketizer = new Bucketizer()
  .setInputCol("CRSDepTime")
  .setOutputCol("bucketedFeatures")
  .setSplits(Array(0, 600, 1200, 1800, 2400))

// +----------+----------------+
// |CRSDepTime|bucketedFeatures|
// +----------+----------------+
// |       745|             1.0|
// |      1053|             1.0|
// |      1915|             3.0|
// |      1755|             2.0|
// |       832|             1.0|
// |       630|             1.0|
// |       820|             1.0|
// |       945|             1.0|
// |      1245|             2.0|
// |      1645|             2.0|
// +----------+----------------+
// only showing top 10 rows

Typically it will be coupled with OneHotEncoder:

import org.apache.spark.ml.Pipeline

val encoder = new OneHotEncoder()
  .setInputCol(bucketizer.getOutputCol)
  .setOutputCol("CRSDepTimeencoded")

val pipeline = new Pipeline().setStages(Array(bucketizer, encoder))

pipeline.fit(df).transform(df).show(10)

// +----------+----------------+-----------------+
// |CRSDepTime|bucketedFeatures|CRSDepTimeencoded|
// +----------+----------------+-----------------+
// |       745|             1.0|    (3,[1],[1.0])|
// |      1053|             1.0|    (3,[1],[1.0])|
// |      1915|             3.0|        (3,[],[])|
// |      1755|             2.0|    (3,[2],[1.0])|
// |       832|             1.0|    (3,[1],[1.0])|
// |       630|             1.0|    (3,[1],[1.0])|
// |       820|             1.0|    (3,[1],[1.0])|
// |       945|             1.0|    (3,[1],[1.0])|
// |      1245|             2.0|    (3,[2],[1.0])|
// |      1645|             2.0|    (3,[2],[1.0])|
// +----------+----------------+-----------------+
// only showing top 10 rows

Turning a continuous variable into categorical in Spark

1 Answers1