Databricks - Creating permanent User Defined Functions (UDFs)

Question

I am able to create a UDF function and register to spark using spark.UDF method. However, this is per session only. How to register python UDF functions automatically when the Cluster starts?. These functions should be available to all users. Example use case is to convert time from UTC to local time zone.

thebluephantom · Accepted Answer · 2019-02-15T21:20:07.290

4

This is not possible; this is not like UDFs in Hive.

Code the UDF as part of the package / program you submit or in the jar included in the Spark App, if using spark-submit.

However,

spark.udf.register.udf("...

is required to be done as well. This applies to Databrick notebooks, etc. The UDFs need to be re-registered per Spark Context/Session.

edited Feb 15 '19 at 21:20

answered Feb 15 '19 at 18:37

thebluephantom

16,458
8
40
83

Thanks. This helps. I will create a notebook with the common functions and call that in master notebook to register the functions. – Sam Feb 18 '19 at 16:53

score 3 · Answer 2 · answered May 10 '19 at 11:00

3

acutally you can create a permanent function but not from a notebook you need to create it from a JAR file

https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-function.html

CREATE [TEMPORARY] FUNCTION [db_name.]function_name AS class_name [USING resource, ...]

resource: : (JAR|FILE|ARCHIVE) file_uri

answered May 10 '19 at 11:00

Gerhard Brueckl

708
1
9
24

Is it possible to create permanent function using python file rather than jar file? – Shanil Apr 06 '23 at 08:33
as stated in the answer below - you can using `spark.udf.register.udf(...)` but you have to do it for each session again – Gerhard Brueckl Apr 09 '23 at 09:07

Databricks - Creating permanent User Defined Functions (UDFs)

2 Answers2

Linked