Spark SQL DF - How to pass multiple values dynamically for `isin` method of `Column`

Question

I am trying to filter the Spark SQL DataFrame with where condition.

For example: [ working one]

df.where(col("pType").isin("type1","type2"))

What I am trying is, instead of hard coding the pType values, I am trying to build it dynamically. So i build a List(String) with all the pType values.

For example, if I have List("type1","type2"), how can I use this list to isin method?

see also http://stackoverflow.com/questions/32551919/spark-filter-isin-doesnt-work-as-expected — Raphael Roth, Oct 26 '16 at 07:58

score 3 · Accepted Answer · edited Jan 07 '19 at 15:41

3

You can use Scala's syntax for converting a collection into a "repeated parameter" (AKA "varargs" in Java-speak, as well as other languages), See section 4.6.2 in the Scala Language Specification:

val list = List("type1","type2")
df.where(col("pType").isin(list: _*))

edited Jan 07 '19 at 15:41

zero323

322,348
103
959
935

answered Oct 25 '16 at 07:09

Tzach Zohar

37,442
3
79
85

Spark SQL DF - How to pass multiple values dynamically for `isin` method of `Column`

1 Answers1