NameError: name 'split' is not defined with Spark

Question

I have been working on a big dataset with Spark. Last week when I ran the following lines of code it worked perfectly, now it is throwing an error: NameError: name 'split' is not defined. Can somebody explain why this is not working and what should I do? Name split is not defined... Should I define the method? Is it a dependency that I should import? The documentation doesn't say I ahve to import anything in order to use the split method. The code below.

test_df = spark_df.withColumn(
  "Keywords", 
   split(col("Keywords"), "\\|")
)

score 6 · Accepted Answer · edited Jan 22 '20 at 23:32

6

You can use pyspark.sql.functions.split(), but you first need to import this function:

from pyspark.sql.functions import split

It's better to explicitly import just the functions you need. Do not do from pyspark.sql.functions import *.

edited Jan 22 '20 at 23:32

pault

41,343
15
107
149

answered Dec 11 '19 at 18:59

werner

13,518
6
30
45

Same error: "NameError: name 'split' is not defined" – Chique_Code Dec 11 '19 at 19:02
1

Which version of pyspark do you use? – werner Dec 11 '19 at 19:04
I am using '2.4.4' – Chique_Code Dec 11 '19 at 19:05
2

Could you please try the import statement `from pyspark.sql.functions import *`? – werner Dec 11 '19 at 19:10

NameError: name 'split' is not defined with Spark

1 Answers1