Scala filter multiple condition

Question

How do I perform a DataFrame select for Distance greater than or equal to 50 AND a Treatment_Type of 1 or 2? My code below seems not to like the multiple condition &&/and in the filter statement.

df.select("Distance", "Treatment_Type").filter("Distance >= 50").filter("Treatment_Type == 1" && "Treatment_Type == 2").show

Thanks in advance.

You mentioned Treatment_Type as 1 or 2 but your code has it as &&. Please use || and let us know. — Amit, Feb 20 '20 at 13:23
you can use this: `df.select("Distance", "Treatment_Type").filter(df.col("Distance") >= 50 && df.col("Treatment_Type") == 1 && df.col("Treatment_Type") == 2).show()` — salmaZ, Feb 20 '20 at 13:24
Does this answer your question? [multiple conditions for filter in spark data frames](https://stackoverflow.com/questions/35881152/multiple-conditions-for-filter-in-spark-data-frames) — user10938362, Feb 20 '20 at 13:48

Lamanus · Accepted Answer · 2020-02-21T08:09:06.623

You want OR condition but you gave the condition for the Treatment_Type 1 AND 2. So, you should give the correct OR condition. Here is an example dataframe

+---+--------+--------------+
| id|Distance|Treatment_Type|
+---+--------+--------------+
|  1|      40|             1|
|  2|      40|             2|
|  3|      40|             3|
|  4|      50|             1|
|  5|      50|             2|
|  6|      50|             3|
|  7|      60|             1|
|  8|      60|             2|
+---+--------+--------------+

Under the below conditions, it give the same results as follows:

df.select("Distance", "Treatment_Type").filter(col("Distance") >= 50 && (col("Treatment_Type") === 1 || col("Treatment_Type") === 2))
df.select("Distance", "Treatment_Type").filter(col("Distance") >= 50 && col("Treatment_Type").isin(1,2))
df.select("Distance", "Treatment_Type").where("Distance >= 50 and Treatment_Type in (1, 2)").show()

+--------+--------------+
|Distance|Treatment_Type|
+--------+--------------+
|      50|             1|
|      50|             2|
|      60|             1|
|      60|             2|
+--------+--------------+

Scala filter multiple condition

1 Answers1