PySpark: How to fillna values in dataframe for specific columns?

Question

I have the following sample DataFrame:

a    | b    | c   | 

1    | 2    | 4   |
0    | null | null| 
null | 3    | 4   |

And I want to replace null values only in the first 2 columns - Column "a" and "b":

a    | b    | c   | 

1    | 2    | 4   |
0    | 0    | null| 
0    | 3    | 4   |

Here is the code to create sample dataframe:

rdd = sc.parallelize([(1,2,4), (0,None,None), (None,3,4)])
df2 = sqlContext.createDataFrame(rdd, ["a", "b", "c"])

I know how to replace all null values using:

df2 = df2.fillna(0)

And when I try this, I lose the third column:

df2 = df2.select(df2.columns[0:1]).fillna(0)

You can find more options here: https://stackoverflow.com/a/65811297/38368 — Danny Varod, Jan 05 '22 at 19:57

score 142 · Accepted Answer · edited Apr 17 '19 at 20:08

142

df.fillna(0, subset=['a', 'b'])

There is a parameter named subset to choose the columns unless your spark version is lower than 1.3.1

edited Apr 17 '19 at 20:08

Endre Both

answered Jul 13 '17 at 01:49

Zhang Tong

wouldn't this give a new df with only one column? anyway to do this in place – Fizi Mar 01 '21 at 23:28
2

@Fizi in Spark, Dataframe is immutable, which means it's impossible to do this in place. – johnnyasd12 Nov 16 '21 at 03:13

score 73 · Answer 2 · answered May 14 '18 at 20:26

73

Use a dictionary to fill values of certain columns:

df.fillna( { 'a':0, 'b':0 } )

answered May 14 '18 at 20:26

scottlittle

1

This is a better answer because it does not matter wether it is one or many values being filled in. – Chris Marotta Jun 17 '20 at 19:25
@ChrisMarotta Does the values type of all selected columns have to be of same type? Could it also be possible: `df.fillna( { 'a':0, 'b':'2022-12-01' } )` where column a is of numeric type, and b is of date type? – nam Jun 05 '22 at 23:09
1

@nam, I suggest you fire up a pyspark terminal and find out – Chris Marotta Jun 16 '22 at 16:36
the behavious that @nam asked for is possible. see the third example in https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrameNaFunctions.fill.html – Naveen Reddy Marthala Jul 26 '23 at 10:15

2 Answers2