can't define a udf inside pyspark project

Question

I have a python project that uses pyspark and i am trying to define a udf function inside the spark project (not in my python project) specifically in spark\python\pyspark\ml\tuning.py but i get pickling problems. it can't load the udf. The code:

from pyspark.sql.functions import udf, log
test_udf = udf(lambda x : -x[1], returnType=FloatType())
d = data.withColumn("new_col", test_udf(data["x"]))
d.show()

when i try d.show() i am getting exception of unknown attribute test_udf

In my python project i defined many udf and it worked fine.

Is this just a typo in entering this into stack overflow, or is this in your code as well? `ltest_udf(` should be `test_udf(` right? — user3124181, Sep 22 '16 at 14:35
Try this, remove `returnType=`, just write it as `udf(lambda x : -x, FloatType())` — user3124181, Sep 22 '16 at 14:48

score 0 · Answer 1 · answered Sep 22 '16 at 18:23

0

add the following to your code. It isn't recognizing the datatype.

from pyspark.sql.types import *

Let me know if this helps. Thanks.

answered Sep 22 '16 at 18:23

avrsanjay

805
7
12

It didn't help. see my answer above but thanks anyway – ofer-a Sep 22 '16 at 19:01
Oh i see, I tried it in the shell and it wasn't recognizing the data type. So had to add that import. thanks for sharing the answer. – avrsanjay Sep 23 '16 at 01:46

score 0 · Answer 2 · answered Sep 22 '16 at 19:01

Found it there was 2 problems

1) for some reason it didn't like the returnType=FloatType() i needed to convert it to just FloatType() though this was the signature

2) The data in column x was a vector and for some reason i had to cast it to float

The working code:

from pyspark.sql.functions import udf, log
test_udf = udf(lambda x : -float(x[1]), FloatType())
d = data.withColumn("new_col", test_udf(data["x"]))
d.show()

can't define a udf inside pyspark project

2 Answers2