7

Does anyone have experience with debugging Pyspark that runs on AWS EMR using Pycharm?

I couldn't find any good guides or existing threads regrading this one.

I know how to debug Scala-Spark with Intellij against EMR but I have no experince with doing this with Python.

I am aware of being able to connect to remote server using ssh (EMR Master) and maybe with Professional edition I can use the remote deployment feature to run my spark job using Pycharm but I'm not sure if it will work and I want to know if anyone has tried it, before I will go with Pycharm Pro.

mck
  • 40,932
  • 13
  • 35
  • 50
Ron F
  • 370
  • 2
  • 14

1 Answers1

1

I got to debug Pyspark on EMR as I wanted to. Please look at this Medium blog post that describes how to do so:

https://medium.com/explorium-ai/debugging-pyspark-with-pycharm-and-aws-emr-d50f90077c92

It describes how to use Pycharm Pro - remote deployment feature in order to debug your pyspark program.

Ron F
  • 370
  • 2
  • 14