check if a row value is null in spark dataframe

Question

I am using a custom function in pyspark to check a condition for each row in a spark dataframe and add columns if condition is true.

The code is as below:

from pyspark.sql.types import *
from pyspark.sql.functions import *
from pyspark.sql import Row

def customFunction(row):
    if (row.prod.isNull()):
        prod_1 = "new prod"
        return (row + Row(prod_1))
    else:
        prod_1 = row.prod
        return (row + Row(prod_1))

sdf = sdf_temp.map(customFunction)
sdf.show()

I get the error mention below:

AttributeError: 'unicode' object has no attribute 'isNull'

How can I check for null values for specific columns in the current row in my custom function?

Could you `show` your `Dataframe` or at least print its schema? — Alberto Bonsanto, Aug 19 '16 at 11:10
Schema of Dataframe is: root |-- id: string (nullable = true) |-- code: string (nullable = true) |-- prod_code: string (nullable = true) |-- prod: string (nullable = true) — sam, Sep 06 '16 at 09:09

score 15 · Answer 1 · answered Sep 06 '16 at 17:10

15

Considering that sdf is a DataFrame you can use a select statement.

sdf.select("*", when(col("pro").isNull(), lit("new pro")).otherwise(col("pro")))

answered Sep 06 '16 at 17:10

Alberto Bonsanto

17,556
10
64
93

But I need to do several operations on different columns of the dataframe, hence wanted to use a custom function. Why can I check for nulls in custom function? – sam Sep 07 '16 at 05:07
1

You need to modify the question, and add your requirements. – Alberto Bonsanto Sep 07 '16 at 10:44

check if a row value is null in spark dataframe

1 Answers1