How to use the withcolumn method with a filter based on more than condition?

Question

I am new to Pyspark

I have this piece of code:

df2 = df.withColumn("VALUE", F.when(col('DIFF') < -900000, None).otherwise(col('VALUE')))

Is it possible to add another condition in the when clause, something like:

df2 = df.withColumn("VALUE", F.when(col('DIFF') < -900000 | col('DIFF') > 900000, None).otherwise(col('VALUE')))

However, this throws an Method does not exist error.

Any ideas?

Thanks in advance.

Can you add the error message? – P.Panayotov Jan 09 '20 at 13:09 — P.Panayotov, Jan 09 '20 at 13:09

score 2 · Accepted Answer · answered Jan 09 '20 at 13:03

2

Like so:

df2 = df.withColumn("VALUE", F.when((col('DIFF') < -900000 ) | (col('DIFF') > 900000), None).otherwise(col('VALUE')))

answered Jan 09 '20 at 13:03

MGJ-123

1 Answers1