Debugging a Livy job

Asked Jun 25 '19 at 14:10

Active Jun 25 '19 at 14:36

Viewed 586 times

I have a PySpark job that I submit to Livy by using the Python client.

I would like to debug it, and I've come across this article, but it's about Java and uses the configuration value of spark.driver.extraJavaOptions, while I need it to be in Python.

So is there a way to debug it as it gets executed (via pdb or any other method)?

edited Jun 25 '19 at 14:36

asked Jun 25 '19 at 14:10

Bolchojeet

Take a look at [How can PySpark be called in debug mode?](https://stackoverflow.com/q/31245083) – user10938362 Jun 25 '19 at 14:14
@user10938362 are you sure that that thread applies to Livy, too? – Bolchojeet Jun 25 '19 at 14:25
Let's say I don't see any reason why it wouldn't. – user10938362 Jun 25 '19 at 14:27

Debugging a Livy job

0 Answers0