Using variable in expression with Nullif fails

Question

I'm using the code below to convert an empty string to Null. I don't get an error but it remains an empty string.
I suspect an incorrect use of the variable col_name in expr

for col_name in ['col1', 'col2']:
    df_new = df \
        .withColumn(col_name, F.expr(f"nullif('{col_name}', '')"))

I get an error `col should be Column` `.withColumn(col_name, f"nullif({col_name}, '')")` — John Doe, Jun 21 '21 at 10:53
but the dataframe will not be updated using your code, see the answer below — mck, Jun 21 '21 at 11:54

score 1 · Accepted Answer · answered Jun 21 '21 at 11:06

What you really want to do is this. Be aware that the for-loop will create and overwrite df_new, so the final column only be changed.

from pyspark.sql import functions as f

df_new = df
for col_name in ['col1', 'col2']:
    df_new = df_new.withColumn(col_name, f.expr(f"nullif({col_name}, '')"))

Using variable in expression with Nullif fails

1 Answers1