0

I am New to Pyspark world. I understand That Encoders are used by Spark SQl to serialize deserialize objects internally by Spark SQL. I know it is about converting Objects to spark sql internal storage types. I see lot of examples on internet about Encoders in spark scala world. I donot see any example in Pyspark world. Moreover , I could not find Encoders in any of Pyspark imports

eg.

from pyspark.sql.catalyst.encoders import RowEncoder

Does not exist in spark

infact catalyst package does not exist in pyspark. Can anyone explain why Encoders are not meaningful in Pyspark?

Harjeet Kumar
  • 504
  • 2
  • 7
  • 1
    See [Why is no map function for dataframe in pyspark while the spark equivalent has it?](https://stackoverflow.com/questions/47343815/why-is-no-map-function-for-dataframe-in-pyspark-while-the-spark-equivalent-has-i) – 10465355 Jan 16 '19 at 09:28
  • 1
    God bless you @user10465355 I was looking for it. – eliasah Jan 16 '19 at 09:42
  • Thanks For comment, Linked question specifically talks about why map is supported on df in scala and why it si not supported in pyspark. Can you elaborate and answer how it is connected with Encoders – Harjeet Kumar Jan 16 '19 at 10:40
  • @eliasah Hi, Please explain how linked question is related to this question. as My question is about encoders. excuse me if i am missing something basic here – Harjeet Kumar Jan 18 '19 at 09:00
  • There is no place for strongly typed DS thus no need for those Encoder like answered in the other question. I'm not sure what is not clear here tbh. – eliasah Jan 18 '19 at 09:57

0 Answers0