Check null column value and replace it with a CONSTANT in Spark DataFrame

Question

I am new to Spark and Scala.

I have created a DataFrame from a csv in Spark. There is a column in the generated DataFrame which has a null value for some rows.

I want to check for this null value and replace it with a constant word say "Hello".

How can I do this in Spark?

Here is my sample code to create a dataframe from csv.

val DFCsv = spark.read.format("csv") .option("sep", ',') .option("inferSchema", "true") .option("header", "true") .load("/tmp/my.csv")
 DFCsv.show()

Now one of the columns in that dataframe named "id" is null or empty/blank for some of the rows.

How can I iterate over each row one by one and then fill out column named "id" with a constant "Hello" String.

Possible duplicate of [Replace null values in Spark DataFrame](https://stackoverflow.com/questions/33376571/replace-null-values-in-spark-dataframe) — user10938362, Jan 29 '19 at 12:37

score -2 · Accepted Answer · answered Jan 29 '19 at 12:38

-2

One of the easiest way is to surround the null value with a Option and then do a pattern match on it.

Option(null) gets converted to None
Option(null).getOrElse("Hello)

answered Jan 29 '19 at 12:38

Chaitanya

Here is my sample code to create a dataframe from csv. val DFCsv = spark.read.format("csv") .option("sep", ',') .option("inferSchema", "true") .option("header", "true") .load("/tmp/my.csv") DFCsv.show() Now one of the columns in that dataframe named "id" is null or empty/blank for some of the rows.How can I iterate over each row one by one and then fill out column named "id" with a constant "Hello" String. – user1393608 Jan 30 '19 at 03:36

1 Answers1