2

From this JIRA ticket Hide UserDefinedType in Spark 2.0 , seems that spark hide the UDT API from version 2.0.

Is there exists an alternative function or API we can use in version 2.2, so that we could define UserDefinedType? I wish to use a custom type in dataframe or structured streaming.

xuanyue
  • 1,368
  • 1
  • 17
  • 36

1 Answers1

4

There is no alternative API and UDT remains private (https://issues.apache.org/jira/browse/SPARK-7768).

Generic Encoders (org.apache.spark.sql.Encoders.kryo and org.apache.spark.sql.Encoders.javaSerialization) serve similar purpose in Dataset, but there are not direct replacement:

Alper t. Turker
  • 34,230
  • 9
  • 83
  • 115