Array within an array of Vectors.dense in Spark

Question

I am trying to add featureD as an array of Double in Vectors.dense function, but getting this error:

templates/scala-parallel-classification/reading-custom-properties/src/main/scala/DataSource.scala:58:21: overloaded method value dense with alternatives:
[INFO] [Engine$] [error]   (values: Array[Double])org.apache.spark.mllib.linalg.Vector <and>
[INFO] [Engine$] [error]   (firstValue: Double,otherValues: Double*)org.apache.spark.mllib.linalg.Vector
[INFO] [Engine$] [error]  cannot be applied to (Array[Any])
[INFO] [Engine$] [error]             Vectors.dense(Array(

This is my code:

required = Some(List( // MODIFIED
    "featureA", "featureB", "featureC", "featureD", "label")))(sc)
  // aggregateProperties() returns RDD pair of
  // entity ID and its aggregated properties
  .map { case (entityId, properties) =>
    try {
      // MODIFIED
      LabeledPoint(properties.get[Double]("label"),
        Vectors.dense(Array(
          properties.get[Double]("featureA"),
          properties.get[Double]("featureB"),
          properties.get[Double]("featureC"),
          properties.get[Array[Double]]("featureD")
        ))
      )
    } catch {
      case e: Exception => {
        logger.error(s"Failed to get properties ${properties} of" +
          s" ${entityId}. Exception: ${e}.")
        throw e
      }
    }

How can I pass array within an array of Vectors.dense function?

score 0 · Answer 1 · answered May 02 '19 at 02:46

Vectors.dense will only accept a single Array[Double] or doubles as separate arguments. It's not possible to have an array within an array. Since the array have mixed types you get the error message:

cannot be applied to (Array[Any])

To solve this, the solution is to simply extend the array with the second array instead of adding it as a single element. In this case, change the creation of LabeledPoint to be:

LabeledPoint(properties.get[Double]("label"),
  Vectors.dense(
    Array(
      properties.get[Double]("featureA"),
      properties.get[Double]("featureB"),
      properties.get[Double]("featureC")
    ) ++ properties.get[Array[Double]]("featureD")
  )
)

Array within an array of Vectors.dense in Spark

1 Answers1