0

does Spark 2.4.0 support Python UDFs with Continuous Processing mode?

In my simple code i'm consuming from a kafka topic, doing some trivial processing per-row (basically add a dummy field to the json messages) and write out to another topic.

However i dont see any messages in the output topic when i trigger using continuous mode. If i dont use the python udf, then it works fine.

EDIT: This bug report (unrelated to this post) reported lack of support.

Venki
  • 417
  • 4
  • 7

1 Answers1

0

Although I have not tried myself, I believe not. The documentation lists the supported operations; udfs are not on the list.

Matthew Jackson
  • 199
  • 2
  • 12