0

My data looks like below 

+------------+--------------+---------------+    
|domain      | country_code | country       | 
+------------+--------------+---------------+
|amazon.de   | DE           | Germany       |
|amazon.uk   | UK           | united kingdom|
|amazon.de   | UK           | mismatched    |
|amazon.uk   | DE           | mismatched    |
+------------+--------------+---------------+

In the above data I want to correct the country_code so anything containing .de in domain column should be checked against a country_code column and if Country_code contains DE then it's a correct match. Anything otherwise is incorrect

So I am trying to create a new column country like below. However, I am unable to use the and statement while using when. Can you please help

import pyspark.sql.functions as f

df = df.withColumn(
    'country', 
    f.when(
        f.col('domain') == '.de' && f.col('country_code') == 'DE',
        'Germany'
    ).otherwise('mismatch')
)
pault
  • 41,343
  • 15
  • 107
  • 149
EricA
  • 403
  • 2
  • 14
  • 1
    Change the double `&` to a single: `f.when((f.col('domain') == '.de') & (f.col('country_code') == 'DE'),'Germany')`. I also generally add parentheses to make it clearer. **Edit**: If you read the [linked dupe](https://stackoverflow.com/questions/37707305/pyspark-multiple-conditions-in-when-clause), you'll see that the parentheses have to be added as well because of operator precedence. – pault Oct 25 '18 at 13:42
  • `&&` is Java, no? Python is just `and`... – SBylemans Oct 25 '18 at 13:43
  • 1
    @SBylemans for this comparison (pyspark columns), you need to use the bitwise and operator (`&`) – pault Oct 25 '18 at 13:44
  • Possible duplicate of [Pyspark: multiple conditions in when clause](https://stackoverflow.com/questions/37707305/pyspark-multiple-conditions-in-when-clause) – pault Oct 25 '18 at 13:45
  • @pault : Thanks. Removing the extra & , parenthesis solved the issue – EricA Oct 25 '18 at 14:18

0 Answers0