Convert values in a single column in pyspark dataframe to lowercase in text cleanup using lower function

Question

Trying to convert convert values in a pyspark dataframe single column to lowercase for the text cleanup using .lower function

import pyspark.sql.functions as f
f.lower(f.col(col("subject")).show()

Getting:

SyntaxError: unexpected EOF while parsing

Answer 1

Your problem is a parenthesis and also you only need to use f.col() once.

import pyspark.sql.functions as f
f.lower(f.col("subject")).show()

You need to assign it to your dataframe:

import pyspark.sql.functions as f
df = df.withColumn("subject",f.lower(f.col("subject")))
df.show()

Answer 2

show is a method for dataframes.

let's assume your dataframe is df, you can do :

df.withColumn(
    "subject", 
    F.lower(F.col("subject"))
).show()