Array_max spark.sql.function not found

Question

I need to use the function array_max and array_min from the package org.apache.spark.sql.functions._ but both functions are not found?

scala> import org.apache.spark.sql.functions._
     import org.apache.spark.sql.functions._
scala>... array_max(col(..))
error: not found: value array_max

ps :

scala version 2.11.8
spark version 2.3.0.2.6.5.0-292
HDP 2.6.5

score 0 · Accepted Answer · answered May 04 '20 at 13:46

0

def array_max(e: org.apache.spark.sql.Column): org.apache.spark.sql.Column

This array_max function is not available in Spark 2.3 & it is only available from Spark 2.4 onwards.

For reference please check below git repos.

Spark 2.3

https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/functions.scala

Spark 2.4

https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/functions.scala

answered May 04 '20 at 13:46

Srinivas

8,957
2
12
26

can we find functions that do the same thing in version 2.3 ? – zekri sidi mohamed hicham May 04 '20 at 14:13
You can write simple udf - val arrayMax = udf((values: Seq[Int]) => values.max); df.select(arrayMax($"value").as("array_max")).show – Srinivas May 04 '20 at 14:22

Array_max spark.sql.function not found

1 Answers1