When executing from within my Jupyter Notebook some cells containing some Spark commands (eg., some DataFrame.show() methods or some spark.sql select commands involving 6 million row DataFrames), I get the following sequence of message errors:
Py4JJavaError: An error occurred while calling xxxx.showString.
SparkException: Job aborted due to stage failure.
Caused by: org.apache.spark.SparkException: Python worker failed to connect back.
Caused by: java.net.SocketTimeoutException: Accept timed out.
How can I interpret them?
I work in a local 8g Spark session.
