1

Is there any method where we can delete the records from a dataframe where any of the column values is null or empty?

+---+-------+--------+-------------------+-----+----------+
|id |zipcode|type    |city               |state|population|
+---+-------+--------+-------------------+-----+----------+
|1  |704    |STANDARD|                   |PR   |30100     |
|2  |704    |        |PASEO COSTA DEL SUR|PR   |          |
|3  |76166  |UNIQUE  |CINGULAR WIRELESS  |TX   |84000     |
+---+-------+--------+-------------------+-----+----------+

I want output to be:

+---+-------+------+-----------------+-----+----------+
|id |zipcode|type  |city             |state|population|
+---+-------+------+-----------------+-----+----------+
|4  |76166  |UNIQUE|CINGULAR WIRELESS|TX   |84000     |
+---+-------+------+-----------------+-----+----------+
zero298
  • 25,467
  • 10
  • 75
  • 100
Shruti Gusain
  • 67
  • 1
  • 6
  • 1
    Try this: `df_name.na.drop() .show(false)` – Mohd Bilal May 14 '20 at 19:08
  • Thanks it worke. Can you please help me on the below mentioned post: It will be a great help. Thanks in advance. https://stackoverflow.com/questions/61815514/remove-null-array-field-from-dataframe-while-converting-it-to-json – Shruti Gusain May 16 '20 at 09:17

2 Answers2

3

Try this:

df
  .na.replace(df.columns,Map("" -> null)) // convert empty strings with null
  .na.drop() // drop nulls and NaNs
  .show()
Raphael Roth
  • 26,751
  • 15
  • 88
  • 145
  • Thanks it worked. Can you help me in the below mentioned post. Thanks in Advance. https://stackoverflow.com/questions/61815514/remove-null-array-field-from-dataframe-while-converting-it-to-json – Shruti Gusain May 16 '20 at 09:18
1

Try this:

df_name.na.drop()
  .show(false)

Hope it helps...

Mohd Bilal
  • 101
  • 8
  • It worked. Can you please help me with this below mentioned post. https://stackoverflow.com/questions/61815514/remove-null-array-field-from-dataframe-while-converting-it-to-json – Shruti Gusain May 16 '20 at 09:18