filter only not empty arrays dataframe spark

Question

How can i filter only not empty arrays

import  org.apache.spark.sql.types.ArrayType

  val arrayFields = secondDF.schema.filter(st => st.dataType.isInstanceOf[ArrayType])
  val names = arrayFields.map(_.name)

Or is this code

val DF1=DF.select(col("key"),explode(col("objectiveAttachment")).as("collection")).select(col("collection.*"),col("key"))

|-- objectiveAttachment: array (nullable = true) 
 | |-- element: string (containsNull = true)

I get this error

 org.apache.spark.sql.AnalysisException: Can only star expand struct data types. Attribute: ArrayBuffer(collection);

Any help is appreciated.

Can you share a reproducible code? – Andronicus Apr 01 '19 at 19:05 — Andronicus, Apr 01 '19 at 19:05

Henrique Florencio · Accepted Answer · 2019-11-12T11:30:00.863

21

Use the function size

import org.apache.spark.sql.functions._

secondDF.filter(size($"objectiveAttachment") > 0)

edited Nov 12 '19 at 11:30

answered Apr 01 '19 at 19:27

Henrique Florencio

3,440
1
18
19

I get error org.apache.spark.sql.AnalysisException: Can only star expand struct data types. Attribute: ArrayBuffer(collection); – Carlos Apr 01 '19 at 19:32
When you use explode, you do not have an array anymore. Try just this code: DF.select(col("key"),explode(col("objectiveAttachment"))) – Henrique Florencio Apr 01 '19 at 19:38
val df= DF.select(col("ins"),explode(col("objectiveAttachment")).printSchema Incomplete expression – Carlos Apr 01 '19 at 19:42
you are missing a parenthesis ) – Henrique Florencio Apr 01 '19 at 19:43
looks like an extra parenthesis to me – user3685285 Nov 05 '19 at 20:19
you are right @user3685285. Just fixed it. Thanks! – Henrique Florencio Nov 12 '19 at 11:30

score 1 · Answer 2 · answered Apr 01 '19 at 19:19

1

Try with size() function from org.apache.spark.sql.functions._

    import org.apache.spark.sql.functions._
    val df1=df.select(col("key"),explode(col("objectiveAttachment")).as("collection")).select(col("collection.*"),col("ins"))
.filter(size($"objectiveAttachment")>0)

answered Apr 01 '19 at 19:19

deo

916
7
8

I get this error org.apache.spark.sql.AnalysisException: Can only star expand struct data types. Attribute: `ArrayBuffer(collection)`; – Carlos Apr 01 '19 at 19:27
My answer was to the filter question, where you can use size() function. Arrays can't be flattened with star operator, you have to use explode() function. – deo Apr 01 '19 at 19:34
Always i get error org.apache.spark.sql.AnalysisException: Can only star expand struct data types. Attribute: ArrayBuffer(collection); – Carlos Apr 01 '19 at 19:42
I mentioned that above. use explode() not star. star operator is only for struct, you are trying to use if for array. – deo Apr 01 '19 at 19:45

filter only not empty arrays dataframe spark

2 Answers2